[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:26 dist-test-slave-gpc1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:26 dist-test-slave-gpc1 krb5kdc[399](info): set up 2 sockets
Sep 25 14:50:26 dist-test-slave-gpc1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:50:27.650626 392 mini_kdc.cc:121] Time spent starting KDC: real 2.570s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:27 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:27.771665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42515
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:27.856796 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:27.857008 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:27.857033 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:27.858263 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:27.858309 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:27.858326 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:27.858402 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:27.858444 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:27.860039 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:27.860302 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:27.860580 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:27.865825 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:27.865965 412 server_base.cc:1034] running on GCE node
W20260925 14:50:27.865813 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:27.865825 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:27.866286 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:27.866689 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:27.867863 412 hybrid_clock.cc:648] HybridClock initialized: now 1790347827867854 us; error 40 us; skew 500 ppm
Sep 25 14:50:27 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:27.874183 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:27.874756 412 webserver.cc:460] Webserver started at http://127.0.98.62:41507/ using document root <none> and password file <none>
I20260925 14:50:27.875005 412 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:27.875073 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:27.875217 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:27.876289 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "14b9eae0a457499a95fe3d89d3927b9b"
format_stamp: "Formatted at 2026-09-25 14:50:27 on dist-test-slave-gpc1"
server_key: "873c9eb034f150f2d9188373937b1f5b"
server_key_iv: "052401de334e3f0a5aa0e1bd580536f2"
server_key_version: "encryptionkey@0"
I20260925 14:50:27.876680 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "14b9eae0a457499a95fe3d89d3927b9b"
format_stamp: "Formatted at 2026-09-25 14:50:27 on dist-test-slave-gpc1"
server_key: "873c9eb034f150f2d9188373937b1f5b"
server_key_iv: "052401de334e3f0a5aa0e1bd580536f2"
server_key_version: "encryptionkey@0"
I20260925 14:50:27.878103 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:27.879037 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:27.879222 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:27.879323 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "14b9eae0a457499a95fe3d89d3927b9b"
format_stamp: "Formatted at 2026-09-25 14:50:27 on dist-test-slave-gpc1"
server_key: "873c9eb034f150f2d9188373937b1f5b"
server_key_iv: "052401de334e3f0a5aa0e1bd580536f2"
server_key_version: "encryptionkey@0"
I20260925 14:50:27.879410 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:27.895120 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:27.897089 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:27.897279 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:27.901523 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42515
I20260925 14:50:27.901559 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42515 every 8 connection(s)
I20260925 14:50:27.901918 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:27.902863 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:27.905156 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b: Bootstrap starting.
I20260925 14:50:27.905679 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:27.905910 480 log.cc:824] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:27.906459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 412
I20260925 14:50:27.906560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:27.906697 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b: No bootstrap required, opened a new log
I20260925 14:50:27.907334 392 external_mini_cluster.cc:1428] Setting key ad16b49a1edb7ad8f332a959b9513571
I20260925 14:50:27.908933 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } }
I20260925 14:50:27.909071 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:27.909093 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b9eae0a457499a95fe3d89d3927b9b, State: Initialized, Role: FOLLOWER
I20260925 14:50:27.909235 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } }
I20260925 14:50:27.909314 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:27.909337 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:27.909404 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:27.910219 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } }
I20260925 14:50:27.910302 480 leader_election.cc:302] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14b9eae0a457499a95fe3d89d3927b9b; no voters:
I20260925 14:50:27.910490 480 leader_election.cc:288] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:27.910581 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:27.910741 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:27.910758 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [term 1 LEADER]: Becoming Leader. State: Replica: 14b9eae0a457499a95fe3d89d3927b9b, State: Running, Role: LEADER
I20260925 14:50:27.910979 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } }
I20260925 14:50:27.911305 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b9eae0a457499a95fe3d89d3927b9b. Latest consensus state: current_term: 1 leader_uuid: "14b9eae0a457499a95fe3d89d3927b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } } }
I20260925 14:50:27.911381 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:27.911592 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:27.912190 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:27.912254 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14b9eae0a457499a95fe3d89d3927b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b9eae0a457499a95fe3d89d3927b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42515 } } }
I20260925 14:50:27.912325 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:27.914422 494 catalog_manager.cc:1347] Generated new cluster ID: 07288cea96f24eb8a17207d92b74e0bf
I20260925 14:50:27.914474 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:27 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:27.924568 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:27.908262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.0.98.62:42515)
0925 14:50:27.908441 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:27.908451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:27.908716 (+ 265us) server_negotiation.cc:407] Connection header received
0925 14:50:27.910585 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:27.910592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:27.910648 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:27.910789 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:27.912975 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:27.913993 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:27.915488 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:27.916080 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:27.916181 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:27.920385 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:27.920389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:27.920390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:27.920401 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:27.922391 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:27.922882 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:27.922886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:27.922886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:27.923011 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:27.923280 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:27.923281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:27.923282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:27.923581 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:27.923745 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:27.924001 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 14:50:27.924163 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260925 14:50:27.930446 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:27.930926 494 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:27.936371 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14b9eae0a457499a95fe3d89d3927b9b: Generated new TSK 0
I20260925 14:50:27.936580 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:27.966857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:28.050194 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:28.050423 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:28.050448 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:28.051653 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:28.051697 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:28.051750 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:28.053182 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:28.053424 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:28.053671 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:28.058564 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:28.058555 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:28.058586 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:28.058830 506 server_base.cc:1034] running on GCE node
I20260925 14:50:28.059041 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:28.059303 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:28.060477 506 hybrid_clock.cc:648] HybridClock initialized: now 1790347828060456 us; error 32 us; skew 500 ppm
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:28.062959 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:28.063436 506 webserver.cc:460] Webserver started at http://127.0.98.1:38177/ using document root <none> and password file <none>
I20260925 14:50:28.063670 506 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:28.063737 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:28.063884 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:28.064913 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "c71fad02e1884c5bb67ea31a5b69799a"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "eacddaba676591f37a2968bdcf723db6"
server_key_iv: "785685f40f259f8cd052245dabb6c1e9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.065300 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c71fad02e1884c5bb67ea31a5b69799a"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "eacddaba676591f37a2968bdcf723db6"
server_key_iv: "785685f40f259f8cd052245dabb6c1e9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.066663 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:28.067391 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.067570 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:28.067664 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "c71fad02e1884c5bb67ea31a5b69799a"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "eacddaba676591f37a2968bdcf723db6"
server_key_iv: "785685f40f259f8cd052245dabb6c1e9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.067746 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:28.076327 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:28.078082 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:28.078238 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:28.078541 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:28.078832 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:28.078887 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.078929 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:28.078984 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.084903 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46795
I20260925 14:50:28.085008 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46795 every 8 connection(s)
I20260925 14:50:28.085299 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:28.091888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 506
I20260925 14:50:28.091977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:28.092131 392 external_mini_cluster.cc:1428] Setting key c0e7f0904d4fbbd950034297e558179c
I20260925 14:50:28.096319 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.086007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38871 (local address 127.0.98.62:42515)
0925 14:50:28.086048 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:28.086050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:28.086258 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:50:28.086859 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:28.086860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:28.086875 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:28.086921 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:28.088401 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.088915 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.089980 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.090189 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.090265 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.092604 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:28.092606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:28.092606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:28.092621 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:28.094850 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.095346 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.095347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.095348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.095401 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.095683 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.095683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.095684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.095782 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:28.095832 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:28.096180 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 14:50:28.096254 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:28.096582 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.085988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42515 (local address 127.0.98.1:38871)
0925 14:50:28.086139 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:28.086173 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:28.086752 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:28.087002 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:28.087006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:28.087975 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:28.088295 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.088301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.089043 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.089044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.089870 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.089872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.090271 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.090272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.090428 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.090896 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:28.090903 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:28.092487 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:28.094954 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.094959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.094961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.095264 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.095469 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.095470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.095470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.095621 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.095844 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:28.095849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:28.096035 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 14:50:28.096161 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 14:50:28.096339 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260925 14:50:28.097049 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42515
I20260925 14:50:28.097142 635 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:28.097329 635 heartbeater.cc:511] Master 127.0.98.62:42515 requested a full tablet report, sending...
I20260925 14:50:28.097903 444 ts_manager.cc:194] Registered new tserver with Master: c71fad02e1884c5bb67ea31a5b69799a (127.0.98.1:46795)
I20260925 14:50:28.098579 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38871
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:28.117429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:28.198414 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:28.198649 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:28.198690 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:28.199834 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:28.199901 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:28.199949 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:28.201391 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:28.201663 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:28.201893 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:28.207621 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:28.207698 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:28.207773 640 server_base.cc:1034] running on GCE node
W20260925 14:50:28.207902 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:28.208124 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:28.208380 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:28.209568 640 hybrid_clock.cc:648] HybridClock initialized: now 1790347828209557 us; error 34 us; skew 500 ppm
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:28.212473 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:28.212859 640 webserver.cc:460] Webserver started at http://127.0.98.2:33865/ using document root <none> and password file <none>
I20260925 14:50:28.213112 640 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:28.213160 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:28.213289 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:28.214421 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c116bbfe330495bb9c8241c3a42b08c"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "90fa56d75fc02ee2618312d528c5d7f1"
server_key_iv: "66a57b7c42d4f74858defde969bdfaf7"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.214794 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c116bbfe330495bb9c8241c3a42b08c"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "90fa56d75fc02ee2618312d528c5d7f1"
server_key_iv: "66a57b7c42d4f74858defde969bdfaf7"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.216159 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:28.216929 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.217154 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:28.217244 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "4c116bbfe330495bb9c8241c3a42b08c"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "90fa56d75fc02ee2618312d528c5d7f1"
server_key_iv: "66a57b7c42d4f74858defde969bdfaf7"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.217316 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:28.232486 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:28.234889 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:28.235065 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:28.235313 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:28.235635 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:28.235690 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.235731 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:28.235761 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.242113 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42877
I20260925 14:50:28.242190 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42877 every 8 connection(s)
I20260925 14:50:28.242539 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:28.252269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 640
I20260925 14:50:28.252369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:28.252471 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.243377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34339 (local address 127.0.98.62:42515)
0925 14:50:28.243470 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:28.243472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:28.243619 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:50:28.244230 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:28.244231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:28.244242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:28.244309 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:28.245414 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.245932 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.247179 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.247377 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.247443 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.249661 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:28.249663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:28.249663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:28.249671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:28.251083 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.251485 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.251486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.251487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.251544 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.251773 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.251774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.251774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.251885 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:28.251943 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:28.252249 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 14:50:28.252347 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:50:28.252559 392 external_mini_cluster.cc:1428] Setting key bad07cfd75ea04c84ba938ff02effddb
I20260925 14:50:28.252619 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.243290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42515 (local address 127.0.98.2:34339)
0925 14:50:28.243475 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:28.243518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:28.244023 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:28.244423 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:28.244427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:28.245073 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:28.245298 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.245303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.246029 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.246030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.247075 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.247078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.247491 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.247491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.247598 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.248086 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:28.248095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:28.249567 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:28.251158 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.251164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.251166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.251379 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.251608 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.251609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.251609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.251710 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.251943 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:28.251946 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:28.252063 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 14:50:28.252207 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:50:28.252391 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260925 14:50:28.253340 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42515
I20260925 14:50:28.253448 769 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:28.253631 769 heartbeater.cc:511] Master 127.0.98.62:42515 requested a full tablet report, sending...
I20260925 14:50:28.254079 443 ts_manager.cc:194] Registered new tserver with Master: 4c116bbfe330495bb9c8241c3a42b08c (127.0.98.2:42877)
I20260925 14:50:28.254535 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34339
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:28.277552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:28.358240 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:28.358460 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:28.358484 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:28.359575 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:28.359611 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:28.359637 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:28.361233 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42515
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:28.361495 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:28.361747 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:28.367030 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:28.367066 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:28.367066 774 server_base.cc:1034] running on GCE node
W20260925 14:50:28.367051 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:28.367511 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:28.367784 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:28.368947 774 hybrid_clock.cc:648] HybridClock initialized: now 1790347828368934 us; error 29 us; skew 500 ppm
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:28.371582 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:28.372071 774 webserver.cc:460] Webserver started at http://127.0.98.3:39127/ using document root <none> and password file <none>
I20260925 14:50:28.372292 774 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:28.372361 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:28.372514 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:28.373595 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2c8c90c963947c3b2b1c356c29a6207"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "0e7c119155804ef612fb45885cfa419a"
server_key_iv: "68a707ae982a813ca91def4b0adefbe9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.373967 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2c8c90c963947c3b2b1c356c29a6207"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "0e7c119155804ef612fb45885cfa419a"
server_key_iv: "68a707ae982a813ca91def4b0adefbe9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.375378 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:28.376160 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.376328 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:28.376435 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "e2c8c90c963947c3b2b1c356c29a6207"
format_stamp: "Formatted at 2026-09-25 14:50:28 on dist-test-slave-gpc1"
server_key: "0e7c119155804ef612fb45885cfa419a"
server_key_iv: "68a707ae982a813ca91def4b0adefbe9"
server_key_version: "encryptionkey@0"
I20260925 14:50:28.376511 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:28.386214 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:28.388568 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:28.388728 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:28.388962 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:28.389272 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:28.389328 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.389369 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:28.389396 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:28.396071 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34669
I20260925 14:50:28.396119 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34669 every 8 connection(s)
I20260925 14:50:28.396425 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:28.402235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 774
I20260925 14:50:28.402347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:28.402561 392 external_mini_cluster.cc:1428] Setting key 24563bbb7faa64dc38d16fa276d06bb0
I20260925 14:50:28.405725 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.397108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45639 (local address 127.0.98.62:42515)
0925 14:50:28.397172 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:28.397175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:28.397407 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:50:28.397943 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:28.397944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:28.397957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:28.398006 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:28.399053 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.399556 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.400559 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.400765 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.400834 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.403035 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:28.403036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:28.403036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:28.403044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:28.404613 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.404967 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.404968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.404968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.405020 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.405195 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.405195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.405195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.405292 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:28.405348 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:28.405600 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:50:28.405663 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:50:28.405885 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.397086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42515 (local address 127.0.98.3:45639)
0925 14:50:28.397258 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:28.397318 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:28.397831 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:28.398074 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:28.398078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:28.398745 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:28.398931 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.398936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.399646 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.399647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.400458 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:28.400460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.400852 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.400853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:28.400975 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.401404 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:28.401412 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:28.402943 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:28.404684 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.404688 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.404690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.404900 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.405062 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:28.405063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:28.405063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:28.405146 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:28.405350 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:28.405354 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:28.405466 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 14:50:28.405570 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:50:28.405691 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260925 14:50:28.406322 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42515
I20260925 14:50:28.406477 903 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:28.406657 903 heartbeater.cc:511] Master 127.0.98.62:42515 requested a full tablet report, sending...
I20260925 14:50:28.407017 443 ts_manager.cc:194] Registered new tserver with Master: e2c8c90c963947c3b2b1c356c29a6207 (127.0.98.3:34669)
I20260925 14:50:28.407389 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45639
I20260925 14:50:28.414654 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:28.437409 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.429838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.0.98.1:46795)
0925 14:50:28.429982 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:28.429988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:28.430006 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:50:28.430033 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:28.430035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:28.430048 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:28.430115 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:28.430666 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.431377 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.432211 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.432515 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.432588 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.434837 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:28.434844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:28.434845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:28.434851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:28.436300 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.436633 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.436635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.436636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.436692 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.436862 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.436862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.436863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.437035 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:28.437149 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:28.437212 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:50:28.437274 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260925 14:50:28.437788 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59670
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:28.458974 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:28.452203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.98.1:46795)
0925 14:50:28.452279 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:28.452281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:28.452296 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:50:28.452370 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:28.452371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:28.452383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:28.452434 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:28.452961 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.453441 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.454175 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:28.454417 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:28.454492 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:28.456530 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:28.456531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:28.456532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:28.456538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:28.458000 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.458287 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.458288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.458288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.458337 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:28.458562 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:28.458563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:28.458563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:28.458656 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:28.458742 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:28.458812 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:50:28.458871 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:50:28.459632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 506
I20260925 14:50:28.464334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 640
I20260925 14:50:28.471547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 774
I20260925 14:50:28.477602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 412
2026-09-25T14:50:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3423 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[917](info): set up 2 sockets
Sep 25 14:50:28 dist-test-slave-gpc1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 14:50:29.509855 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:29 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:29.627209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:29.706542 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:29.706806 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:29.706847 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:29.708027 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:29.708098 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:29.708125 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:29.708158 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:29.708189 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:29.709657 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:29.709949 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:29.710183 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:29.715260 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:29.715261 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:29.715332 930 server_base.cc:1034] running on GCE node
W20260925 14:50:29.715313 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:29.715677 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:29.715992 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:29.717167 930 hybrid_clock.cc:648] HybridClock initialized: now 1790347829717154 us; error 45 us; skew 500 ppm
Sep 25 14:50:29 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:29.719668 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:29.720110 930 webserver.cc:460] Webserver started at http://127.0.98.62:39427/ using document root <none> and password file <none>
I20260925 14:50:29.720287 930 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:29.720325 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:29.720409 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:29.721385 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "9fc1c1827601435a8339f0695edb21fc"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "9d27a723fed6dd4ca253d637a06e1b08"
server_key_iv: "0a1ec3ba0be9315ed1db34bea237abbc"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.721701 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fc1c1827601435a8339f0695edb21fc"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "9d27a723fed6dd4ca253d637a06e1b08"
server_key_iv: "0a1ec3ba0be9315ed1db34bea237abbc"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.723012 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:29.723688 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:29.723873 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:29.723954 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "9fc1c1827601435a8339f0695edb21fc"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "9d27a723fed6dd4ca253d637a06e1b08"
server_key_iv: "0a1ec3ba0be9315ed1db34bea237abbc"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.724020 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:29.739368 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:29.741300 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:29.741487 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:29.745718 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33593
I20260925 14:50:29.745781 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33593 every 8 connection(s)
I20260925 14:50:29.746126 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:29.747023 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:29.749609 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc: Bootstrap starting.
I20260925 14:50:29.750224 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:29.750576 998 log.cc:824] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:29.751407 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc: No bootstrap required, opened a new log
I20260925 14:50:29.751655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 930
I20260925 14:50:29.751740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:29.751893 392 external_mini_cluster.cc:1428] Setting key b70d8d09d4fcf7668879fc1d8a443122
I20260925 14:50:29.754295 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260925 14:50:29.754500 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:29.754537 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc1c1827601435a8339f0695edb21fc, State: Initialized, Role: FOLLOWER
I20260925 14:50:29.754689 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260925 14:50:29.754776 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:29.754806 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:29.754850 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:29.755550 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260925 14:50:29.755676 998 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc1c1827601435a8339f0695edb21fc; no voters:
I20260925 14:50:29.755825 998 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:29.755887 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:29.756040 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [term 1 LEADER]: Becoming Leader. State: Replica: 9fc1c1827601435a8339f0695edb21fc, State: Running, Role: LEADER
I20260925 14:50:29.756050 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:29.756141 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } }
I20260925 14:50:29.756543 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc1c1827601435a8339f0695edb21fc. Latest consensus state: current_term: 1 leader_uuid: "9fc1c1827601435a8339f0695edb21fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } } }
I20260925 14:50:29.756547 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc1c1827601435a8339f0695edb21fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc1c1827601435a8339f0695edb21fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33593 } } }
I20260925 14:50:29.756659 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:29.756667 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:29.757588 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:29.758095 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:29.759706 1012 catalog_manager.cc:1347] Generated new cluster ID: 8f64d0965f894ca4bd6746efdefa6277
I20260925 14:50:29.759755 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:29 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:29.764026 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:29.752500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60196 (local address 127.0.98.62:33593)
0925 14:50:29.752698 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:29.752710 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:29.752748 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:50:29.753480 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:29.753498 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:29.753573 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:29.753739 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:29.754286 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.756029 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.757222 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.757913 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.758035 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:29.760678 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:29.760681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:29.760682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:29.760693 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:29.762459 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:29.762869 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:29.762872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:29.762873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:29.762963 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:29.763197 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:29.763198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:29.763198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:29.763436 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:29.763629 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:29.763710 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:50:29.763854 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260925 14:50:29.788504 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:29.789292 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:29.797873 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc1c1827601435a8339f0695edb21fc: Generated new TSK 0
I20260925 14:50:29.798116 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:29.823508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:29.905293 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:29.905517 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:29.905550 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:29.906764 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:29.906804 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:29.906831 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:29.908341 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:29.908612 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:29.908842 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:29.914000 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:29.914014 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:29.914104 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:29.914119 1024 server_base.cc:1034] running on GCE node
I20260925 14:50:29.914435 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:29.914716 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:29.915889 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790347829915870 us; error 40 us; skew 500 ppm
Sep 25 14:50:29 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:29.918605 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:29.919080 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36943/ using document root <none> and password file <none>
I20260925 14:50:29.919312 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:29.919375 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:29.919531 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:29.920636 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d7aa0e9afde42efa9b3561c51da694c"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "ddf36750982c0d50c78ae647825afb67"
server_key_iv: "45e095e52eeaca1cc312351b7a97f1f4"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.920990 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d7aa0e9afde42efa9b3561c51da694c"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "ddf36750982c0d50c78ae647825afb67"
server_key_iv: "45e095e52eeaca1cc312351b7a97f1f4"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.922267 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:29.923141 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:29.923328 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:29.923432 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "6d7aa0e9afde42efa9b3561c51da694c"
format_stamp: "Formatted at 2026-09-25 14:50:29 on dist-test-slave-gpc1"
server_key: "ddf36750982c0d50c78ae647825afb67"
server_key_iv: "45e095e52eeaca1cc312351b7a97f1f4"
server_key_version: "encryptionkey@0"
I20260925 14:50:29.923520 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:29.951412 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:29.953902 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:29.954077 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:29.954329 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:29.954679 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:29.954731 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:29.954773 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:29.954798 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:29.960892 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46697
I20260925 14:50:29.960938 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46697 every 8 connection(s)
I20260925 14:50:29.961302 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:50:29 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:29.968716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1024
I20260925 14:50:29.968808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:29.968993 392 external_mini_cluster.cc:1428] Setting key f7d94d7ab206277aeda0cc6da870d14d
I20260925 14:50:29.970901 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:29.962142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46363 (local address 127.0.98.62:33593)
0925 14:50:29.962186 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:29.962188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:29.962414 (+ 226us) server_negotiation.cc:407] Connection header received
0925 14:50:29.962956 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:29.962957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:29.962971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:29.963027 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:29.963990 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.964511 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.965569 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.965795 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.965881 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:29.968017 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:29.968018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:29.968019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:29.968026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:29.969505 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:29.969896 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:29.969898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:29.969898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:29.969959 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:29.970227 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:29.970228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:29.970228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:29.970379 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:29.970442 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:29.970717 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:50:29.970808 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:29.971264 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:29.962094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.1:46363)
0925 14:50:29.962252 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:29.962304 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:29.962849 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:29.963103 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:29.963107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:29.963691 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:29.963884 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:29.963889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.964630 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.964631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:29.965472 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:29.965475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:29.965892 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:29.965893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:29.966029 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:29.966547 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:29.966556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:29.967931 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:29.969583 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:29.969590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:29.969593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:29.969817 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:29.970024 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:29.970025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:29.970025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:29.970151 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:29.970432 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:29.970438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:29.970584 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:50:29.970696 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:50:29.970859 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260925 14:50:29.971771 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260925 14:50:29.971891 1153 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:29.972045 1153 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
I20260925 14:50:29.972515 962 ts_manager.cc:194] Registered new tserver with Master: 6d7aa0e9afde42efa9b3561c51da694c (127.0.98.1:46697)
I20260925 14:50:29.973145 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46363
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:29.993701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:30.076215 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:30.076447 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:30.076472 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:30.077680 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:30.077729 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:30.077759 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:30.079380 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:30.079654 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:30.079912 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:30.085157 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:30.085250 1158 server_base.cc:1034] running on GCE node
W20260925 14:50:30.085136 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:30.085155 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:30.085572 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:30.085817 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:30.086987 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790347830086979 us; error 40 us; skew 500 ppm
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:30.089576 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:30.090030 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35659/ using document root <none> and password file <none>
I20260925 14:50:30.090260 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:30.090327 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:30.090507 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:30.091593 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "44e7abfe48f042f3b3a98ee94b3a2a57"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "ab1a33791a95f18f2f04da1701f76e4c"
server_key_iv: "34481093b4cdcb95129563d9171e2856"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.091953 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44e7abfe48f042f3b3a98ee94b3a2a57"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "ab1a33791a95f18f2f04da1701f76e4c"
server_key_iv: "34481093b4cdcb95129563d9171e2856"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.093307 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:30.094022 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.094197 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:30.094296 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "44e7abfe48f042f3b3a98ee94b3a2a57"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "ab1a33791a95f18f2f04da1701f76e4c"
server_key_iv: "34481093b4cdcb95129563d9171e2856"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.094408 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:30.110816 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:30.113389 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:30.113581 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:30.113859 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:30.114208 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:30.114266 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.114320 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:30.114370 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.120286 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39621
I20260925 14:50:30.120332 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39621 every 8 connection(s)
I20260925 14:50:30.120699 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:30.128872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1158
I20260925 14:50:30.128968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:30.129163 392 external_mini_cluster.cc:1428] Setting key 8130195330bfdba5052ef03d2bdd4466
I20260925 14:50:30.131318 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.121580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37591 (local address 127.0.98.62:33593)
0925 14:50:30.121671 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.121674 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.121844 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:50:30.122449 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.122451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.122463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.122513 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.123517 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.124031 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.125065 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.125266 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.125351 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.128048 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.128049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.128050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.128060 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.129837 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.130268 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.130269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.130270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.130337 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.130607 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.130608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.130609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.130763 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.130831 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.131087 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.131185 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:50:30.131484 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.121447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.2:37591)
0925 14:50:30.121696 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.121744 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.122290 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.122618 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.122627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.123205 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:30.123402 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.123408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.124139 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.124140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.124970 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.124972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.125360 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.125361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.125498 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.126119 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:30.126128 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:30.127940 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:30.129914 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:30.129922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:30.129925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:30.130190 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:30.130422 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:30.130422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:30.130423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:30.130539 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:30.130826 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:30.130830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:30.130958 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.131042 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.131180 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260925 14:50:30.131975 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260925 14:50:30.132093 1287 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:30.132297 1287 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
I20260925 14:50:30.132694 962 ts_manager.cc:194] Registered new tserver with Master: 44e7abfe48f042f3b3a98ee94b3a2a57 (127.0.98.2:39621)
I20260925 14:50:30.133091 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37591
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:30.155015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:30.236146 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:30.236362 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:30.236387 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:30.237519 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:30.237551 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:30.237576 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:30.239008 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33593
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:30.239192 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:30.239409 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:30.244513 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:30.244527 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:30.244534 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:30.244583 1292 server_base.cc:1034] running on GCE node
I20260925 14:50:30.244983 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:30.245184 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:30.246326 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790347830246294 us; error 46 us; skew 500 ppm
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:30.248984 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:30.249428 1292 webserver.cc:460] Webserver started at http://127.0.98.3:45897/ using document root <none> and password file <none>
I20260925 14:50:30.249686 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:30.249755 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:30.249905 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:30.251148 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "daaa234cfea6410da5945b3fa928b9fe"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "65224e20d8024110728b32b30a9dba9a"
server_key_iv: "35e3b6536a61dddd920499c42eaa5df9"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.251552 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "daaa234cfea6410da5945b3fa928b9fe"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "65224e20d8024110728b32b30a9dba9a"
server_key_iv: "35e3b6536a61dddd920499c42eaa5df9"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.252861 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:30.253602 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.253772 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:30.253873 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "daaa234cfea6410da5945b3fa928b9fe"
format_stamp: "Formatted at 2026-09-25 14:50:30 on dist-test-slave-gpc1"
server_key: "65224e20d8024110728b32b30a9dba9a"
server_key_iv: "35e3b6536a61dddd920499c42eaa5df9"
server_key_version: "encryptionkey@0"
I20260925 14:50:30.253957 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:30.273377 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:30.275519 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:30.275688 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:30.275938 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:30.276260 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:30.276312 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.276352 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:30.276382 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:30.282199 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34523
I20260925 14:50:30.282243 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34523 every 8 connection(s)
I20260925 14:50:30.282604 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:30.290330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1292
I20260925 14:50:30.290441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:30.290612 392 external_mini_cluster.cc:1428] Setting key 4f08640af2286b3a58a1189920b790b0
I20260925 14:50:30.292128 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.283317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52685 (local address 127.0.98.62:33593)
0925 14:50:30.283389 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.283391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.283607 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:50:30.284128 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.284129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.284142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.284207 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.285150 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.285662 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.286832 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.287063 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.287148 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.289125 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.289126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.289126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.289134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.290818 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.291209 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.291210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.291210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.291265 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.291518 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.291519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.291519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.291633 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.291707 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.291960 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.292035 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:50:30.292323 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.283312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33593 (local address 127.0.98.3:52685)
0925 14:50:30.283482 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.283518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.284022 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.284283 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.284287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.284839 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:30.285042 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.285047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.285757 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.285758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.286720 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.286722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.287156 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.287157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.287279 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.287712 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:30.287720 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:30.289032 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:30.290903 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:30.290910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:30.290913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:30.291133 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:30.291339 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:30.291340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:30.291340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:30.291443 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:30.291698 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:30.291703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:30.291834 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.291949 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.292099 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260925 14:50:30.292794 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33593
I20260925 14:50:30.292887 1421 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:30.293074 1421 heartbeater.cc:511] Master 127.0.98.62:33593 requested a full tablet report, sending...
I20260925 14:50:30.293427 962 ts_manager.cc:194] Registered new tserver with Master: daaa234cfea6410da5945b3fa928b9fe (127.0.98.3:34523)
I20260925 14:50:30.293804 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52685
I20260925 14:50:30.302207 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:30.309448 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.303841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.0.98.62:33593)
0925 14:50:30.303904 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.303906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.304016 (+ 110us) server_negotiation.cc:407] Connection header received
0925 14:50:30.304054 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.304054 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.304063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.304115 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.304695 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.305194 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.305874 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.306071 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.306126 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.307286 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.307287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.307288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.307294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.308485 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.308792 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.308793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.308793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.308844 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.309026 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.309026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.309027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.309169 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.309245 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.309328 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.309386 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:50:30.312009 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:30.313791 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:30.320436 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.0.98.3:34523)
0925 14:50:30.316951 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.316956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.316965 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:30.317002 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.317004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.317019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.317094 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:30.317204 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.318214 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.319326 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320021 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320096 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320152 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.320305 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.320356 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260925 14:50:30.320622 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46697 (local address 127.0.0.1:48958)
0925 14:50:30.316572 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.316584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.316833 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.317009 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.317009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.317011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:30.317050 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.317052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.318095 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.318096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.319610 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.319611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320334 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.320386 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320390 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.320418 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.320451 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":63,"thread_start_us":68,"threads_started":1}
I20260925 14:50:30.320698 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39621 (local address 127.0.0.1:38448)
0925 14:50:30.316412 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.316491 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.316555 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.316795 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.316798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.316800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:30.316872 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.316873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.318222 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.318222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.319470 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.319473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320452 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.320539 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320544 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.320594 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.320631 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":55,"thread_start_us":63,"threads_started":1}
I20260925 14:50:30.320721 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.98.1:46697)
0925 14:50:30.316860 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.316864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.316874 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:30.316913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.316915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.316932 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.317013 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:30.317236 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.317995 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.319689 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320245 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320492 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320567 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.320592 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.320629 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":94,"threads_started":1}
I20260925 14:50:30.320426 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34523 (local address 127.0.0.1:45094)
0925 14:50:30.316669 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.316678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.316723 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.317115 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.317116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.317118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:30.317146 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.317148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.318303 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.318305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.319236 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.319239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320109 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.320183 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320198 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.320281 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.320340 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260925 14:50:30.321157 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.316430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.0.98.2:39621)
0925 14:50:30.316618 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.316623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.316633 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:30.316673 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.316675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.316690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.316780 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:30.316950 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.318120 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.319579 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.320301 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.320366 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.320417 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.321024 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.321071 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260925 14:50:30.321494 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 4013df8e24314c64a0bf8e795fa2497a (DEFAULT_TABLE table=test-table [id=a6cbb7a13048442999513b6f7d3aa83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:30.321768 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 4013df8e24314c64a0bf8e795fa2497a (DEFAULT_TABLE table=test-table [id=a6cbb7a13048442999513b6f7d3aa83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:30.321873 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4013df8e24314c64a0bf8e795fa2497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:30.322098 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4013df8e24314c64a0bf8e795fa2497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:30.322751 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 4013df8e24314c64a0bf8e795fa2497a (DEFAULT_TABLE table=test-table [id=a6cbb7a13048442999513b6f7d3aa83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:30.323077 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4013df8e24314c64a0bf8e795fa2497a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:30.324640 1438 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Bootstrap starting.
I20260925 14:50:30.325187 1439 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Bootstrap starting.
I20260925 14:50:30.325491 1438 tablet_bootstrap.cc:654] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:30.325829 1439 tablet_bootstrap.cc:654] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:30.326076 1438 log.cc:824] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:30.326526 1440 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Bootstrap starting.
I20260925 14:50:30.326613 1439 log.cc:824] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:30.327001 1438 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: No bootstrap required, opened a new log
I20260925 14:50:30.327070 1438 ts_tablet_manager.cc:1397] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:50:30.327313 1440 tablet_bootstrap.cc:654] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:30.327384 1439 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: No bootstrap required, opened a new log
I20260925 14:50:30.327459 1439 ts_tablet_manager.cc:1397] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:50:30.328265 1440 log.cc:824] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:30.328933 1440 tablet_bootstrap.cc:492] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: No bootstrap required, opened a new log
I20260925 14:50:30.329018 1440 ts_tablet_manager.cc:1397] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:50:30.328971 1438 raft_consensus.cc:348] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.329178 1438 raft_consensus.cc:374] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:30.329227 1438 raft_consensus.cc:729] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e7abfe48f042f3b3a98ee94b3a2a57, State: Initialized, Role: FOLLOWER
I20260925 14:50:30.329344 1438 consensus_queue.cc:260] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.329622 1438 ts_tablet_manager.cc:1428] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:30.329810 1287 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
I20260925 14:50:30.329865 1439 raft_consensus.cc:348] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.330034 1439 raft_consensus.cc:374] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:30.330084 1439 raft_consensus.cc:729] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7aa0e9afde42efa9b3561c51da694c, State: Initialized, Role: FOLLOWER
I20260925 14:50:30.330207 1439 consensus_queue.cc:260] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.330523 1153 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
I20260925 14:50:30.330724 1439 ts_tablet_manager.cc:1428] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260925 14:50:30.331115 1440 raft_consensus.cc:348] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.331278 1440 raft_consensus.cc:374] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:30.331328 1440 raft_consensus.cc:729] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daaa234cfea6410da5945b3fa928b9fe, State: Initialized, Role: FOLLOWER
I20260925 14:50:30.331437 1440 consensus_queue.cc:260] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.331733 1440 ts_tablet_manager.cc:1428] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:30.331802 1421 heartbeater.cc:503] Master 127.0.98.62:33593 was elected leader, sending a full tablet report...
W20260925 14:50:30.371270 1288 tablet.cc:2369] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:30.462150 1154 tablet.cc:2369] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:30.533178 1422 tablet.cc:2369] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:30.638314 1445 raft_consensus.cc:482] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:30.638506 1445 raft_consensus.cc:504] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.638926 1445 leader_election.cc:288] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e7abfe48f042f3b3a98ee94b3a2a57 (127.0.98.2:39621), daaa234cfea6410da5945b3fa928b9fe (127.0.98.3:34523)
I20260925 14:50:30.641932 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.639161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39199 (local address 127.0.98.3:34523)
0925 14:50:30.639210 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.639212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.639451 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:50:30.639477 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.639478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.639490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.639543 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:30.639717 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.640171 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641164 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.641556 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641651 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.641663 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.641822 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.641862 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:30.641984 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.639111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34523 (local address 127.0.98.1:39199)
0925 14:50:30.639333 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.639342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.639405 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.639553 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.639554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.639555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:30.639645 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.639648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.640270 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.640271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.641067 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.641069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.641655 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.641731 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.641735 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.641810 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.641852 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":63,"threads_started":1}
I20260925 14:50:30.642014 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.639176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39567 (local address 127.0.98.2:39621)
0925 14:50:30.639222 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.639224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.639305 (+ 81us) server_negotiation.cc:407] Connection header received
0925 14:50:30.639358 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.639359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.639373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.639434 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:30.639634 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.640161 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641176 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.641564 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.641672 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.641863 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.641903 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:50:30.642005 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.639095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39621 (local address 127.0.98.1:39567)
0925 14:50:30.639227 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:30.639238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:30.639305 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:30.639447 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:30.639448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:30.639450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:30.639557 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.639560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.640242 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.640244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.641095 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:30.641097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.641665 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.641666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:30.641769 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.641773 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:30.641855 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:50:30.641895 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260925 14:50:30.642433 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4013df8e24314c64a0bf8e795fa2497a" candidate_uuid: "6d7aa0e9afde42efa9b3561c51da694c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" is_pre_election: true
I20260925 14:50:30.642505 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4013df8e24314c64a0bf8e795fa2497a" candidate_uuid: "6d7aa0e9afde42efa9b3561c51da694c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa234cfea6410da5945b3fa928b9fe" is_pre_election: true
I20260925 14:50:30.642623 1241 raft_consensus.cc:2454] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7aa0e9afde42efa9b3561c51da694c in term 0.
I20260925 14:50:30.642671 1375 raft_consensus.cc:2454] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d7aa0e9afde42efa9b3561c51da694c in term 0.
I20260925 14:50:30.642866 1043 leader_election.cc:302] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e7abfe48f042f3b3a98ee94b3a2a57, 6d7aa0e9afde42efa9b3561c51da694c; no voters:
I20260925 14:50:30.643097 1445 raft_consensus.cc:2781] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:30.643146 1445 raft_consensus.cc:482] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:30.643191 1445 raft_consensus.cc:3037] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:30.643898 1445 raft_consensus.cc:504] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.644048 1445 leader_election.cc:288] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [CANDIDATE]: Term 1 election: Requested vote from peers 44e7abfe48f042f3b3a98ee94b3a2a57 (127.0.98.2:39621), daaa234cfea6410da5945b3fa928b9fe (127.0.98.3:34523)
I20260925 14:50:30.644156 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4013df8e24314c64a0bf8e795fa2497a" candidate_uuid: "6d7aa0e9afde42efa9b3561c51da694c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57"
I20260925 14:50:30.644224 1241 raft_consensus.cc:3037] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:30.644200 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4013df8e24314c64a0bf8e795fa2497a" candidate_uuid: "6d7aa0e9afde42efa9b3561c51da694c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daaa234cfea6410da5945b3fa928b9fe"
I20260925 14:50:30.644268 1375 raft_consensus.cc:3037] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:30.644874 1241 raft_consensus.cc:2454] T 4013df8e24314c64a0bf8e795fa2497a P 44e7abfe48f042f3b3a98ee94b3a2a57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7aa0e9afde42efa9b3561c51da694c in term 1.
I20260925 14:50:30.644886 1375 raft_consensus.cc:2454] T 4013df8e24314c64a0bf8e795fa2497a P daaa234cfea6410da5945b3fa928b9fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d7aa0e9afde42efa9b3561c51da694c in term 1.
I20260925 14:50:30.645080 1042 leader_election.cc:302] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d7aa0e9afde42efa9b3561c51da694c, daaa234cfea6410da5945b3fa928b9fe; no voters:
I20260925 14:50:30.645212 1445 raft_consensus.cc:2781] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:30.645381 1445 raft_consensus.cc:686] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [term 1 LEADER]: Becoming Leader. State: Replica: 6d7aa0e9afde42efa9b3561c51da694c, State: Running, Role: LEADER
I20260925 14:50:30.645494 1445 consensus_queue.cc:237] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } }
I20260925 14:50:30.646389 961 catalog_manager.cc:5495] T 4013df8e24314c64a0bf8e795fa2497a P 6d7aa0e9afde42efa9b3561c51da694c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d7aa0e9afde42efa9b3561c51da694c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6d7aa0e9afde42efa9b3561c51da694c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7aa0e9afde42efa9b3561c51da694c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e7abfe48f042f3b3a98ee94b3a2a57" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "daaa234cfea6410da5945b3fa928b9fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34523 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:30.679126 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.671360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.0.98.1:46697)
0925 14:50:30.671433 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.671435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.671491 (+ 56us) server_negotiation.cc:407] Connection header received
0925 14:50:30.671547 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.671547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.671559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.671599 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.672596 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.673079 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.673855 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.674045 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.674100 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.676428 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.676438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.676438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.676444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.678119 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.678450 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.678454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.678454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.678514 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.678670 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.678670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.678670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.678792 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.678888 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.678941 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.679013 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:30.702482 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.695007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48982 (local address 127.0.98.1:46697)
0925 14:50:30.695083 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.695086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.695184 (+ 98us) server_negotiation.cc:407] Connection header received
0925 14:50:30.695282 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.695283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.695293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.695365 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.695927 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.696404 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.697210 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.697428 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.697481 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.699886 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.699888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.699889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.699895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.701255 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.701621 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.701623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.701623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.701679 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.701880 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.701881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.701881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.701979 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.702086 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.702279 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.702381 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:30.724385 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:30.717626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.0.98.1:46697)
0925 14:50:30.717692 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:30.717694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:30.717771 (+ 77us) server_negotiation.cc:407] Connection header received
0925 14:50:30.717858 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:30.717859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:30.717875 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:30.717939 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:30.718571 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.719039 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.719793 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:30.719982 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:30.720064 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:30.722244 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:30.722245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:30.722245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:30.722252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:30.723494 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.723791 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.723792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.723792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.723839 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:30.723992 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:30.723993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:30.723993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:30.724078 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:30.724149 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:30.724230 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:50:30.724283 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 14:50:30.724884 1067 tablet_service.cc:2908] Rejecting scan request for tablet 4013df8e24314c64a0bf8e795fa2497a: Uninitialized: clean time has not yet been initialized
I20260925 14:50:30.725751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1024
I20260925 14:50:30.730686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1158
I20260925 14:50:30.735554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1292
I20260925 14:50:30.740042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 930
2026-09-25T14:50:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2263 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[1463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[1463](info): set up 2 sockets
Sep 25 14:50:30 dist-test-slave-gpc1 krb5kdc[1463](info): commencing operation
krb5kdc: starting...
W20260925 14:50:31.808848 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260925 14:50:32.042613 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.141s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:32.247735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:32.327198 1476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:32.327450 1476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:32.327491 1476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:32.328637 1476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:32.328704 1476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:50:32.328729 1476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:32.328763 1476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:32.328786 1476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:32.328816 1476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:32.330391 1476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:32.330684 1476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:32.330906 1476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:32.335520 1481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.335491 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.335546 1484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.335754 1476 server_base.cc:1034] running on GCE node
I20260925 14:50:32.335986 1476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:32.336228 1476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:32.337368 1476 hybrid_clock.cc:648] HybridClock initialized: now 1790347832337340 us; error 44 us; skew 500 ppm
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:32.339833 1476 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:32.340289 1476 webserver.cc:460] Webserver started at http://127.0.98.62:38135/ using document root <none> and password file <none>
I20260925 14:50:32.340554 1476 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:32.340601 1476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:32.340768 1476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:32.341781 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "1c28329db9584e9e81f4d9f34b4c1f21"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "be3640ade30606369ee3b3a7a5650af7"
server_key_iv: "2dcea8310ec40ec635aeaaff232c9f2a"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.342139 1476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c28329db9584e9e81f4d9f34b4c1f21"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "be3640ade30606369ee3b3a7a5650af7"
server_key_iv: "2dcea8310ec40ec635aeaaff232c9f2a"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.343513 1476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:32.344219 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.344400 1476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:32.344530 1476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "1c28329db9584e9e81f4d9f34b4c1f21"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "be3640ade30606369ee3b3a7a5650af7"
server_key_iv: "2dcea8310ec40ec635aeaaff232c9f2a"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.344609 1476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:32.370947 1476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:32.373292 1476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:32.373466 1476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:32.378345 1476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43393
I20260925 14:50:32.378412 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43393 every 8 connection(s)
I20260925 14:50:32.378741 1476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:32.379340 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:32.381537 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Bootstrap starting.
I20260925 14:50:32.382097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1476
I20260925 14:50:32.382170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:32.382185 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:32.382330 392 external_mini_cluster.cc:1428] Setting key 941c6a87c92c2c1cb4c9998d8f4f20dd
I20260925 14:50:32.382582 1554 log.cc:824] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:32.383365 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: No bootstrap required, opened a new log
I20260925 14:50:32.385811 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } }
I20260925 14:50:32.385979 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:32.386013 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c28329db9584e9e81f4d9f34b4c1f21, State: Initialized, Role: FOLLOWER
I20260925 14:50:32.386133 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } }
I20260925 14:50:32.386225 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:32.386250 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:32.386322 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:32.386866 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } }
I20260925 14:50:32.386953 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c28329db9584e9e81f4d9f34b4c1f21; no voters:
I20260925 14:50:32.387166 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:32.387248 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:32.387389 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:32.387441 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [term 1 LEADER]: Becoming Leader. State: Replica: 1c28329db9584e9e81f4d9f34b4c1f21, State: Running, Role: LEADER
I20260925 14:50:32.387698 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } }
I20260925 14:50:32.388095 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c28329db9584e9e81f4d9f34b4c1f21. Latest consensus state: current_term: 1 leader_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } } }
I20260925 14:50:32.388176 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:32.389436 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:32.389765 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c28329db9584e9e81f4d9f34b4c1f21" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43393 } } }
I20260925 14:50:32.389842 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:32.389916 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:32.391700 1568 catalog_manager.cc:1347] Generated new cluster ID: 80de854a8f664201a167e033604ce32b
I20260925 14:50:32.391763 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:32.394750 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.382918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.0.98.62:43393)
0925 14:50:32.383112 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.383122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.383154 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:50:32.383747 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.383756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.383820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.383972 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.384572 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.386044 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.387023 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.387656 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.388074 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.390726 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.390730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.390730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.390741 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.392744 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.393156 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.393159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.393159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.393252 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.393480 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.393481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.393482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.393767 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.393900 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.394410 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.394548 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260925 14:50:32.421607 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:32.422360 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:32.425803 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Generated new TSK 0
I20260925 14:50:32.425985 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:32.457571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43393
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:32.562033 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:32.562302 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:32.562331 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:32.563537 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:32.563577 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:32.563598 1580 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:32.563616 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:32.565033 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:32.565208 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:32.565378 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:32.570304 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.570379 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.570521 1580 server_base.cc:1034] running on GCE node
W20260925 14:50:32.570302 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.570827 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:32.571105 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:32.572286 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790347832572279 us; error 53 us; skew 500 ppm
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:32.574885 1580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:32.575322 1580 webserver.cc:460] Webserver started at http://127.0.98.1:37659/ using document root <none> and password file <none>
I20260925 14:50:32.575553 1580 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:32.575596 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:32.575723 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:32.576772 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "e64aed07c9434aff88ddaabd9e546c5e"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "e35d177f41fcc677513ac3b72206dfc4"
server_key_iv: "f2f5808eef1f47c92d394a3d5d05862c"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.577132 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e64aed07c9434aff88ddaabd9e546c5e"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "e35d177f41fcc677513ac3b72206dfc4"
server_key_iv: "f2f5808eef1f47c92d394a3d5d05862c"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.578581 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:32.579319 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.579504 1580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:32.579599 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "e64aed07c9434aff88ddaabd9e546c5e"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "e35d177f41fcc677513ac3b72206dfc4"
server_key_iv: "f2f5808eef1f47c92d394a3d5d05862c"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.579679 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:32.595872 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:32.597723 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:32.597886 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:32.598429 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:32.598498 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.598543 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:32.598573 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:32.605621 1580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41177
I20260925 14:50:32.605657 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41177 every 8 connection(s)
I20260925 14:50:32.606015 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:32.608937 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.598923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35807 (local address 127.0.98.62:43393)
0925 14:50:32.598968 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.598970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.599176 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:50:32.599731 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.599732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.599748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.599831 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.600548 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.601441 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.602822 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.603227 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.603313 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.605960 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.605968 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.605969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.605977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.607538 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.607972 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.607973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.607974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.608031 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.608292 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.608293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.608293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.608411 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.608492 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.608763 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.608848 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:32.609149 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.598918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.1:35807)
0925 14:50:32.599068 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.599102 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.599600 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.599922 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.599936 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.600245 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.600437 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.600442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.601535 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.601536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.602743 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.602746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.603337 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.603338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.603551 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.604452 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.604460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.605869 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.607643 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.607650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.607652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.607875 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.608105 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.608106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.608106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.608219 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.608484 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.608490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.608629 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.608737 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.608891 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260925 14:50:32.609684 1711 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43393
I20260925 14:50:32.609795 1711 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:32.609985 1711 heartbeater.cc:511] Master 127.0.98.62:43393 requested a full tablet report, sending...
I20260925 14:50:32.610585 1508 ts_manager.cc:194] Registered new tserver with Master: e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177)
I20260925 14:50:32.611243 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35807
I20260925 14:50:32.613051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1580
I20260925 14:50:32.613125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:32.613320 392 external_mini_cluster.cc:1428] Setting key c9773d556bd6ec5d7b10e99d082cf5ee
I20260925 14:50:32.617947 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.611447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32955 (local address 127.0.98.62:43393)
0925 14:50:32.611520 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.611521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.611530 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:32.611550 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.611551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.611561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.611638 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.612036 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.612580 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.613397 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.613681 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.613770 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.614847 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.614848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.614849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.614855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.616713 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.617119 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.617120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.617121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.617186 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.617412 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.617413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.617413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.617533 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.617615 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.617774 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.617852 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:50:32.617969 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.610735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.1:32955)
0925 14:50:32.610885 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.610896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.610951 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.611643 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.611644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.611872 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.611942 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.611944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.612683 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.612685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.613290 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.613292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.613771 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.613772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.613839 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.614280 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.614285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.614734 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.616827 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.616828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.616829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.617026 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.617267 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.617268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.617268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.617343 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.617605 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.617606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.617651 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.617747 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.617853 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:32.638806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43393
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:32.718001 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:32.718197 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:32.718220 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:32.719398 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:32.719440 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:32.719461 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:32.719481 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:32.720984 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:32.721223 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:32.721444 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:32.726445 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.726553 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.726486 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.726718 1720 server_base.cc:1034] running on GCE node
I20260925 14:50:32.726967 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:32.727224 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:32.728444 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790347832728403 us; error 30 us; skew 500 ppm
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:32.731068 1720 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:32.731544 1720 webserver.cc:460] Webserver started at http://127.0.98.2:44225/ using document root <none> and password file <none>
I20260925 14:50:32.731773 1720 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:32.731817 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:32.731953 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:32.732985 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "a63c8b87c69a06b1aacf86640a18c5f1"
server_key_iv: "c304dd9615aa7ac465afc29aa416fd5d"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.733346 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "a63c8b87c69a06b1aacf86640a18c5f1"
server_key_iv: "c304dd9615aa7ac465afc29aa416fd5d"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.734715 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:32.735464 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.735654 1720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:32.735756 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "a63c8b87c69a06b1aacf86640a18c5f1"
server_key_iv: "c304dd9615aa7ac465afc29aa416fd5d"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.735829 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:32.761554 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:32.764776 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:32.764959 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:32.765471 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:32.765540 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.765586 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:32.765614 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.772182 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44649
I20260925 14:50:32.772300 1850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44649 every 8 connection(s)
I20260925 14:50:32.772622 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:32.773275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1720
I20260925 14:50:32.773365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:32.773535 392 external_mini_cluster.cc:1428] Setting key 8c16a1adecb02c9b80e5ac4e2032efdb
I20260925 14:50:32.776917 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.766005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43387 (local address 127.0.98.62:43393)
0925 14:50:32.766074 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.766077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.766298 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:50:32.766885 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.766886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.766900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.766951 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.767828 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.768698 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.770363 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.770710 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.770794 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.773653 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.773654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.773655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.773663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.775488 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.775904 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.775905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.775905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.775980 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.776222 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.776222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.776223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.776368 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.776438 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.776746 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.776826 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:50:32.777160 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.765990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.2:43387)
0925 14:50:32.766161 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.766202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.766769 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.767023 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.767028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.767409 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.767717 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.767724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.768825 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.768827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.770231 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.770234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.770825 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.770826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.770977 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.771634 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.771646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.773537 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.775596 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.775602 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.775605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.775840 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.776050 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.776051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.776051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.776154 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.776469 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.776475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.776605 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.776703 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.776895 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260925 14:50:32.777598 1851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43393
I20260925 14:50:32.777673 1851 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:32.777848 1851 heartbeater.cc:511] Master 127.0.98.62:43393 requested a full tablet report, sending...
I20260925 14:50:32.778553 1508 ts_manager.cc:194] Registered new tserver with Master: e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649)
I20260925 14:50:32.779104 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43387
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:50:32.784468 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.778722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57981 (local address 127.0.98.62:43393)
0925 14:50:32.778756 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.778758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.778921 (+ 163us) server_negotiation.cc:407] Connection header received
0925 14:50:32.778962 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.778963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.778975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.779035 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.779381 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.780192 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.780862 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.781173 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.781253 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.782107 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.782109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.782109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.782117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.783580 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.783869 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.783870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.783870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.783917 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.784064 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.784065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.784065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.784181 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.784226 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.784329 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.784384 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:32.784458 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.778721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.2:57981)
0925 14:50:32.778841 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.778852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.778910 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.779040 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.779040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.779234 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.779304 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.779307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.780291 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.780291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.780782 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.780783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.781254 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.781255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.781299 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.781752 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.781756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.782019 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.783648 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.783649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.783649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.783812 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.783973 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.783974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.783974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.784023 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.784229 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.784229 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.784263 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.784311 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.784385 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:32.800834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43393
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:32.884809 1860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:32.885020 1860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:32.885043 1860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:32.886195 1860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:32.886237 1860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:32.886257 1860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:32.886276 1860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:32.888005 1860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:32.888252 1860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:32.888509 1860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:32.893513 1865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:32.893560 1868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.893704 1860 server_base.cc:1034] running on GCE node
W20260925 14:50:32.893478 1866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:32.893955 1860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:32.894174 1860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:32.895342 1860 hybrid_clock.cc:648] HybridClock initialized: now 1790347832895328 us; error 25 us; skew 500 ppm
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:32.900750 1860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:32.901237 1860 webserver.cc:460] Webserver started at http://127.0.98.3:42673/ using document root <none> and password file <none>
I20260925 14:50:32.901438 1860 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:32.901479 1860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:32.901569 1860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:32.902674 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "c80c5a7226c44498bca5925dd25c7c40"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "b722b1c2e882d31d9694d5b24f70fb23"
server_key_iv: "6fd081b372cc9f810227847fec2aa9b9"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.902993 1860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c80c5a7226c44498bca5925dd25c7c40"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "b722b1c2e882d31d9694d5b24f70fb23"
server_key_iv: "6fd081b372cc9f810227847fec2aa9b9"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.904443 1860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:32.905167 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.905356 1860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:32.905439 1860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "c80c5a7226c44498bca5925dd25c7c40"
format_stamp: "Formatted at 2026-09-25 14:50:32 on dist-test-slave-gpc1"
server_key: "b722b1c2e882d31d9694d5b24f70fb23"
server_key_iv: "6fd081b372cc9f810227847fec2aa9b9"
server_key_version: "encryptionkey@0"
I20260925 14:50:32.905500 1860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:32.921396 1860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:32.923584 1860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:32.923696 1860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:32.924216 1860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:32.924255 1860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.924279 1860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:32.924292 1860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:32.931154 1860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44955
I20260925 14:50:32.931242 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44955 every 8 connection(s)
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:32.931545 1860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:32.934302 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.924695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42503 (local address 127.0.98.62:43393)
0925 14:50:32.924766 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.924769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.924926 (+ 157us) server_negotiation.cc:407] Connection header received
0925 14:50:32.925477 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.925478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.925491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.925591 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.926569 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.927399 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.928966 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.929283 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.929370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.931869 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.931871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.931872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.931878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.933194 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.933560 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.933561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.933562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.933612 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.933813 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.933814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.933814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.933908 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.933960 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.934175 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.934233 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:50:32.934515 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.924652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.3:42503)
0925 14:50:32.924794 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.924833 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.925359 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.925765 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.925769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.926147 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.926464 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.926471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.927511 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.927512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.928868 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.928872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.929368 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.929369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.929469 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.930072 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.930081 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.931744 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.933267 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.933272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.933274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.933494 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.933668 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.933669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.933669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.933761 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.933961 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.933965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.934070 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.934150 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.934261 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 14:50:32.935051 1991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43393
I20260925 14:50:32.935161 1991 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:32.935335 1991 heartbeater.cc:511] Master 127.0.98.62:43393 requested a full tablet report, sending...
I20260925 14:50:32.935721 1507 ts_manager.cc:194] Registered new tserver with Master: c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3:44955)
I20260925 14:50:32.936147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 1860
I20260925 14:50:32.936177 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42503
I20260925 14:50:32.936257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:32.936456 392 external_mini_cluster.cc:1428] Setting key 9d089be8c2a8f937bcbeff98655ad109
I20260925 14:50:32.937757 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:32.942378 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.936251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36031 (local address 127.0.98.62:43393)
0925 14:50:32.936288 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.936291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.936620 (+ 329us) server_negotiation.cc:407] Connection header received
0925 14:50:32.936699 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.936700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.936711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.936763 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.937091 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.937557 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.938322 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.938649 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.938725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.939755 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.939756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.939757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.939765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.941196 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.941572 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.941573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.941573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.941621 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.941827 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.941828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.941828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.941982 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.942045 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.942224 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.942278 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:50:32.942642 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.936175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.98.3:36031)
0925 14:50:32.936551 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.936562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.936646 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.936770 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.936771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.936956 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:32.937009 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.937011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.937675 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.937676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.938236 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.938238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.938737 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.938738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.938851 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.939336 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:32.939342 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:32.939678 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:32.941267 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.941268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.941269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.941499 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.941687 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:32.941688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:32.941689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:32.941766 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:32.942063 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:32.942064 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:32.942116 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.942205 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.942312 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":156,"threads_started":1}
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:50:32 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:32.976182 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.963050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.0.98.62:43393)
0925 14:50:32.963135 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.963137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.963150 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:50:32.963184 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.963186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.963200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.963287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:32.964287 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.965198 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.965907 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.966224 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.966298 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.973522 (+ 7224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:32.973524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:32.973524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:32.973531 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:32.974904 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.975303 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.975304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.975304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.975364 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:32.975612 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:32.975613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:32.975614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:32.975736 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:32.975839 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.976043 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.976111 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:50:32.978039 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:32.979825 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:32.986536 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44649 (local address 127.0.0.1:36348)
0925 14:50:32.982282 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.982322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.982447 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.982849 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.982852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.982855 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:32.982926 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.982930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.984283 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.984284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.985202 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.985204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.986131 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.986133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.986227 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.986241 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.986322 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.986413 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":35,"thread_start_us":64,"threads_started":1}
I20260925 14:50:32.986564 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.0.98.2:44649)
0925 14:50:32.982647 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.982652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.982664 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:50:32.982707 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.982710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.982727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.983006 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:32.983070 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.984225 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.985284 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.986023 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.986109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.986188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.986420 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.986464 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":78,"threads_started":1}
I20260925 14:50:32.986793 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.0.98.1:41177)
0925 14:50:32.982674 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.982678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.982687 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:32.982720 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.982722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.982735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.982817 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:32.983006 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.984119 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.985369 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.986023 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.986104 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.986190 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.986669 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.986706 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260925 14:50:32.986536 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41177 (local address 127.0.0.1:57498)
0925 14:50:32.982510 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.982520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.982561 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.982887 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.982887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.982889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:32.982926 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.982928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.984202 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.984203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.985295 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.985298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.986133 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.986133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.986225 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.986241 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.986322 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.986413 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":9,"thread_start_us":70,"threads_started":1}
I20260925 14:50:32.987344 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44955 (local address 127.0.0.1:41528)
0925 14:50:32.982630 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:32.982639 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:32.982681 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:32.982998 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:32.983000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:32.983002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:32.983060 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.983062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.983961 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.983962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.985296 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:32.985298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.987097 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.987098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:32.987160 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.987164 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:32.987214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:50:32.987252 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":51,"thread_start_us":102,"threads_started":1}
I20260925 14:50:32.987596 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:32.982488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.98.3:44955)
0925 14:50:32.982637 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:32.982643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:32.982675 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:50:32.982743 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:32.982745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:32.982760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:32.982935 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:32.983122 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.983875 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.986225 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:32.987013 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:32.987082 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:32.987160 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:32.987435 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:50:32.987487 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260925 14:50:32.987723 1785 tablet_service.cc:1467] Processing CreateTablet for tablet b3de2304cf0945d2824abcf655555764 (DEFAULT_TABLE table=test-table [id=b214e346898b49af942388789648485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:32.988137 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3de2304cf0945d2824abcf655555764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:32.990996 1645 tablet_service.cc:1467] Processing CreateTablet for tablet b3de2304cf0945d2824abcf655555764 (DEFAULT_TABLE table=test-table [id=b214e346898b49af942388789648485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:32.990998 1924 tablet_service.cc:1467] Processing CreateTablet for tablet b3de2304cf0945d2824abcf655555764 (DEFAULT_TABLE table=test-table [id=b214e346898b49af942388789648485e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:32.991361 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3de2304cf0945d2824abcf655555764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:32.991361 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3de2304cf0945d2824abcf655555764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:32.991395 2014 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Bootstrap starting.
I20260925 14:50:32.992014 2014 tablet_bootstrap.cc:654] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:32.992336 2014 log.cc:824] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:32.993021 2014 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: No bootstrap required, opened a new log
I20260925 14:50:32.993109 2014 ts_tablet_manager.cc:1397] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:32.994541 2014 raft_consensus.cc:348] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.994695 2014 raft_consensus.cc:374] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:32.994737 2014 raft_consensus.cc:729] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cc5d2c2b9c4475936b756bbdd6b1aa, State: Initialized, Role: FOLLOWER
I20260925 14:50:32.994871 2014 consensus_queue.cc:260] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.995060 2016 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Bootstrap starting.
I20260925 14:50:32.995147 2014 ts_tablet_manager.cc:1428] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:50:32.995270 1851 heartbeater.cc:503] Master 127.0.98.62:43393 was elected leader, sending a full tablet report...
I20260925 14:50:32.995563 2017 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Bootstrap starting.
I20260925 14:50:32.995749 2016 tablet_bootstrap.cc:654] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:32.996068 2016 log.cc:824] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:32.996356 2017 tablet_bootstrap.cc:654] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:32.996695 2017 log.cc:824] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:32.996964 2016 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: No bootstrap required, opened a new log
I20260925 14:50:32.997038 2016 ts_tablet_manager.cc:1397] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:32.997517 2017 tablet_bootstrap.cc:492] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: No bootstrap required, opened a new log
I20260925 14:50:32.997601 2017 ts_tablet_manager.cc:1397] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:32.998653 2016 raft_consensus.cc:348] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.998792 2016 raft_consensus.cc:374] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:32.998816 2016 raft_consensus.cc:729] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80c5a7226c44498bca5925dd25c7c40, State: Initialized, Role: FOLLOWER
I20260925 14:50:32.998945 2016 consensus_queue.cc:260] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.999238 2016 ts_tablet_manager.cc:1428] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:50:32.999281 1991 heartbeater.cc:503] Master 127.0.98.62:43393 was elected leader, sending a full tablet report...
I20260925 14:50:32.999188 2017 raft_consensus.cc:348] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.999364 2017 raft_consensus.cc:374] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:32.999408 2017 raft_consensus.cc:729] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64aed07c9434aff88ddaabd9e546c5e, State: Initialized, Role: FOLLOWER
I20260925 14:50:32.999529 2017 consensus_queue.cc:260] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:32.999835 2017 ts_tablet_manager.cc:1428] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:50:32.999856 1711 heartbeater.cc:503] Master 127.0.98.62:43393 was elected leader, sending a full tablet report...
W20260925 14:50:33.023207 1852 tablet.cc:2369] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:33.106777 1712 tablet.cc:2369] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:33.182132 1992 tablet.cc:2369] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:33.194764 2021 raft_consensus.cc:482] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:33.194890 2021 raft_consensus.cc:504] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:33.195209 2021 leader_election.cc:288] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649), e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177)
I20260925 14:50:33.198437 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.195643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45589 (local address 127.0.98.1:41177)
0925 14:50:33.195699 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.195700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.195779 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:50:33.195840 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.195841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.195853 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.195906 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:33.196127 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.196649 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.197664 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.198010 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.198099 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.198109 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.198266 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.198312 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:50:33.198442 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.195594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36157 (local address 127.0.98.2:44649)
0925 14:50:33.195648 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.195651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.195664 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:50:33.195701 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.195703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.195719 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.195783 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:33.195987 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.196547 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.197558 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.198007 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.198098 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.198109 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.198298 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.198364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:50:33.198477 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.195475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44649 (local address 127.0.98.3:36157)
0925 14:50:33.195540 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.195555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.195624 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.195797 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.195798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.195800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:33.195884 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.195887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.196638 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.196640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.197450 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.197452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.198095 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.198096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.198199 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.198202 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.198273 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.198314 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":15}
I20260925 14:50:33.198576 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.195500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41177 (local address 127.0.98.3:45589)
0925 14:50:33.195690 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.195702 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.195772 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.195945 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.195946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.195948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:33.196024 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.196025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.196745 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.196746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.197579 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.197581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.198103 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.198104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.198193 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.198197 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.198242 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.198306 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260925 14:50:33.198860 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3de2304cf0945d2824abcf655555764" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64aed07c9434aff88ddaabd9e546c5e" is_pre_election: true
I20260925 14:50:33.198886 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3de2304cf0945d2824abcf655555764" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" is_pre_election: true
I20260925 14:50:33.199046 1665 raft_consensus.cc:2454] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 0.
I20260925 14:50:33.199061 1805 raft_consensus.cc:2454] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 0.
I20260925 14:50:33.199297 1879 leader_election.cc:302] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80c5a7226c44498bca5925dd25c7c40, e64aed07c9434aff88ddaabd9e546c5e; no voters:
I20260925 14:50:33.199465 2021 raft_consensus.cc:2781] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:33.199527 2021 raft_consensus.cc:482] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:33.199560 2021 raft_consensus.cc:3037] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.200511 2021 raft_consensus.cc:504] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:33.200675 2021 leader_election.cc:288] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 election: Requested vote from peers e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649), e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177)
I20260925 14:50:33.200819 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3de2304cf0945d2824abcf655555764" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64aed07c9434aff88ddaabd9e546c5e"
I20260925 14:50:33.200819 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3de2304cf0945d2824abcf655555764" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
I20260925 14:50:33.200886 1805 raft_consensus.cc:3037] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.200906 1665 raft_consensus.cc:3037] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.201608 1805 raft_consensus.cc:2454] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 1.
I20260925 14:50:33.201601 1665 raft_consensus.cc:2454] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 1.
I20260925 14:50:33.201776 1879 leader_election.cc:302] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80c5a7226c44498bca5925dd25c7c40, e64aed07c9434aff88ddaabd9e546c5e; no voters:
I20260925 14:50:33.201898 2021 raft_consensus.cc:2781] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:33.202067 2021 raft_consensus.cc:686] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 1 LEADER]: Becoming Leader. State: Replica: c80c5a7226c44498bca5925dd25c7c40, State: Running, Role: LEADER
I20260925 14:50:33.202171 2021 consensus_queue.cc:237] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } }
I20260925 14:50:33.203135 1508 catalog_manager.cc:5495] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 reported cstate change: term changed from 0 to 1, leader changed from <none> to c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c80c5a7226c44498bca5925dd25c7c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:33.219885 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.217373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.98.3:44955)
0925 14:50:33.217455 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.217457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.217465 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:33.217569 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.217570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.217586 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.217645 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:33.217826 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.218270 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.219128 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.219313 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.219369 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.219578 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.219665 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.219724 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.219771 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.219788 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:33.221597 1805 raft_consensus.cc:1261] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Refusing update from remote peer c80c5a7226c44498bca5925dd25c7c40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:33.221895 1665 raft_consensus.cc:1261] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Refusing update from remote peer c80c5a7226c44498bca5925dd25c7c40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:33.222023 2021 consensus_queue.cc:1035] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.222199 2024 consensus_queue.cc:1035] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.228183 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:33.228294 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:33.229015 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Sending DeleteTablet for 3 replicas of tablet b3de2304cf0945d2824abcf655555764
I20260925 14:50:33.229365 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet b3de2304cf0945d2824abcf655555764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260925 14:50:33.229369 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet b3de2304cf0945d2824abcf655555764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36348
I20260925 14:50:33.229601 2036 tablet_replica.cc:330] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: stopping tablet replica
I20260925 14:50:33.229712 2036 raft_consensus.cc:2229] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:33.229840 2036 raft_consensus.cc:2258] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.229820 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet b3de2304cf0945d2824abcf655555764 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260925 14:50:33.229983 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:33.230129 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:33.230314 2036 ts_tablet_manager.cc:1905] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.231619 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 (DEFAULT_TABLE table=test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:33.231654 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 (DEFAULT_TABLE table=test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:33.231752 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a825cf786d448b88d546ca3d68d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.231769 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a825cf786d448b88d546ca3d68d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.232120 2036 ts_tablet_manager.cc:1918] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:33.232180 2036 log.cc:1196] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/b3de2304cf0945d2824abcf655555764
I20260925 14:50:33.232424 2036 ts_tablet_manager.cc:1939] T b3de2304cf0945d2824abcf655555764 P c80c5a7226c44498bca5925dd25c7c40: Deleting consensus metadata
I20260925 14:50:33.232826 1494 catalog_manager.cc:4841] TS c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3:44955): tablet b3de2304cf0945d2824abcf655555764 (table test-table [id=b214e346898b49af942388789648485e]) successfully deleted
I20260925 14:50:33.233039 2014 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Bootstrap starting.
I20260925 14:50:33.233281 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 (DEFAULT_TABLE table=test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:33.233384 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a825cf786d448b88d546ca3d68d2e94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.233453 2014 tablet_bootstrap.cc:654] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.234172 2014 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: No bootstrap required, opened a new log
I20260925 14:50:33.234236 2014 ts_tablet_manager.cc:1397] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:33.234444 2017 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Bootstrap starting.
I20260925 14:50:33.234448 2014 raft_consensus.cc:348] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.234539 2014 raft_consensus.cc:374] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.234562 2014 raft_consensus.cc:729] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cc5d2c2b9c4475936b756bbdd6b1aa, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.234611 2014 consensus_queue.cc:260] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.234704 2014 ts_tablet_manager.cc:1428] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:33.234829 2017 tablet_bootstrap.cc:654] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.234966 2016 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Bootstrap starting.
I20260925 14:50:33.235505 2038 tablet_replica.cc:330] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: stopping tablet replica
I20260925 14:50:33.235525 2016 tablet_bootstrap.cc:654] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.235661 2017 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: No bootstrap required, opened a new log
I20260925 14:50:33.235733 2017 ts_tablet_manager.cc:1397] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:33.235900 2037 tablet_replica.cc:330] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: stopping tablet replica
I20260925 14:50:33.235944 2017 raft_consensus.cc:348] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.236083 2037 raft_consensus.cc:2229] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:33.236227 2037 raft_consensus.cc:2258] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.236449 2017 raft_consensus.cc:374] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.236490 2017 raft_consensus.cc:729] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64aed07c9434aff88ddaabd9e546c5e, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.236546 2017 consensus_queue.cc:260] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.236691 2017 ts_tablet_manager.cc:1428] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:33.236971 2037 ts_tablet_manager.cc:1905] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.235658 2038 raft_consensus.cc:2229] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:33.237749 2038 raft_consensus.cc:2258] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.238165 2016 tablet_bootstrap.cc:492] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: No bootstrap required, opened a new log
I20260925 14:50:33.238219 2016 ts_tablet_manager.cc:1397] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:50:33.238214 2038 ts_tablet_manager.cc:1905] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.238400 2016 raft_consensus.cc:348] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.238499 2016 raft_consensus.cc:374] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.238523 2016 raft_consensus.cc:729] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80c5a7226c44498bca5925dd25c7c40, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.238592 2016 consensus_queue.cc:260] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.238706 2016 ts_tablet_manager.cc:1428] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:33.239480 2037 ts_tablet_manager.cc:1918] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:33.239559 2037 log.cc:1196] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/b3de2304cf0945d2824abcf655555764
I20260925 14:50:33.239846 2037 ts_tablet_manager.cc:1939] T b3de2304cf0945d2824abcf655555764 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting consensus metadata
I20260925 14:50:33.240370 1494 catalog_manager.cc:4841] TS e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649): tablet b3de2304cf0945d2824abcf655555764 (table test-table [id=b214e346898b49af942388789648485e]) successfully deleted
I20260925 14:50:33.240548 2038 ts_tablet_manager.cc:1918] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:33.240621 2038 log.cc:1196] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/b3de2304cf0945d2824abcf655555764
I20260925 14:50:33.241223 2038 ts_tablet_manager.cc:1939] T b3de2304cf0945d2824abcf655555764 P e64aed07c9434aff88ddaabd9e546c5e: Deleting consensus metadata
I20260925 14:50:33.241611 1495 catalog_manager.cc:4841] TS e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177): tablet b3de2304cf0945d2824abcf655555764 (table test-table [id=b214e346898b49af942388789648485e]) successfully deleted
I20260925 14:50:33.385247 2021 raft_consensus.cc:482] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:33.385330 2021 raft_consensus.cc:504] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.385499 2021 leader_election.cc:288] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177), e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649)
I20260925 14:50:33.385746 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a825cf786d448b88d546ca3d68d2e94" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64aed07c9434aff88ddaabd9e546c5e" is_pre_election: true
I20260925 14:50:33.385746 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a825cf786d448b88d546ca3d68d2e94" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" is_pre_election: true
I20260925 14:50:33.385828 1665 raft_consensus.cc:2454] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 0.
I20260925 14:50:33.385856 1805 raft_consensus.cc:2454] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 0.
I20260925 14:50:33.386015 1879 leader_election.cc:302] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80c5a7226c44498bca5925dd25c7c40, e64aed07c9434aff88ddaabd9e546c5e; no voters:
I20260925 14:50:33.386132 2021 raft_consensus.cc:2781] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:33.386183 2021 raft_consensus.cc:482] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:33.386196 2021 raft_consensus.cc:3037] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.386842 2021 raft_consensus.cc:504] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.386974 2021 leader_election.cc:288] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 election: Requested vote from peers e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177), e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649)
I20260925 14:50:33.387117 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a825cf786d448b88d546ca3d68d2e94" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
I20260925 14:50:33.387151 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a825cf786d448b88d546ca3d68d2e94" candidate_uuid: "c80c5a7226c44498bca5925dd25c7c40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e64aed07c9434aff88ddaabd9e546c5e"
I20260925 14:50:33.387215 1805 raft_consensus.cc:3037] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.387229 1665 raft_consensus.cc:3037] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.387946 1805 raft_consensus.cc:2454] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 1.
I20260925 14:50:33.387993 1665 raft_consensus.cc:2454] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c80c5a7226c44498bca5925dd25c7c40 in term 1.
I20260925 14:50:33.388152 1878 leader_election.cc:302] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c80c5a7226c44498bca5925dd25c7c40, e5cc5d2c2b9c4475936b756bbdd6b1aa; no voters:
I20260925 14:50:33.388249 2021 raft_consensus.cc:2781] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:33.388342 2021 raft_consensus.cc:686] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 1 LEADER]: Becoming Leader. State: Replica: c80c5a7226c44498bca5925dd25c7c40, State: Running, Role: LEADER
I20260925 14:50:33.388396 2021 consensus_queue.cc:237] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.389024 1508 catalog_manager.cc:5495] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 reported cstate change: term changed from 0 to 1, leader changed from <none> to c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c80c5a7226c44498bca5925dd25c7c40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } health_report { overall_health: HEALTHY } } }
Sep 25 14:50:33 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:33.457988 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.451094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43393 (local address 127.0.0.1:49964)
0925 14:50:33.451236 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.451245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.451351 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.451506 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.451506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.451835 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:33.451880 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.451883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.452530 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.452531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.453124 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.453126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.453487 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.453488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.453535 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.454128 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:33.454133 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:33.455711 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:33.457067 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:33.457074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:33.457076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:33.457266 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:33.457461 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:33.457462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:33.457462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:33.457518 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:33.457726 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:33.457732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:33.457776 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.457830 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.457915 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260925 14:50:33.458050 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.451132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.98.62:43393)
0925 14:50:33.451193 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.451195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.451350 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:50:33.451415 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.451416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.451426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.451501 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:33.451978 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.452452 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.453198 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.453408 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.453488 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.455788 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:33.455789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:33.455790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:33.455795 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:33.456991 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.457330 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.457331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.457331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.457377 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.457576 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.457576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.457576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.457668 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:33.457715 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.457859 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.457969 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":34}
I20260925 14:50:33.460012 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:50:33.460197 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:33.461530 1785 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1838134564ba98e4ff66671db2e8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d595e6d2a544e0b2fd79a652351061]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:33.461573 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1838134564ba98e4ff66671db2e8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d595e6d2a544e0b2fd79a652351061]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:33.461565 1924 tablet_service.cc:1467] Processing CreateTablet for tablet c9c1838134564ba98e4ff66671db2e8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94d595e6d2a544e0b2fd79a652351061]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:33.461696 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1838134564ba98e4ff66671db2e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.461696 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1838134564ba98e4ff66671db2e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.461716 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c1838134564ba98e4ff66671db2e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:33.462909 2016 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: Bootstrap starting.
I20260925 14:50:33.463220 2014 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: Bootstrap starting.
I20260925 14:50:33.463297 2016 tablet_bootstrap.cc:654] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.463603 2014 tablet_bootstrap.cc:654] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.463886 2016 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: No bootstrap required, opened a new log
I20260925 14:50:33.463937 2016 ts_tablet_manager.cc:1397] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:33.463963 2017 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: Bootstrap starting.
I20260925 14:50:33.464041 2016 raft_consensus.cc:348] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.464138 2016 raft_consensus.cc:374] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.464157 2016 raft_consensus.cc:729] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80c5a7226c44498bca5925dd25c7c40, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.464268 2014 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: No bootstrap required, opened a new log
I20260925 14:50:33.464303 2014 ts_tablet_manager.cc:1397] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:33.464416 2014 raft_consensus.cc:348] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.464466 2014 raft_consensus.cc:374] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.464479 2014 raft_consensus.cc:729] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5cc5d2c2b9c4475936b756bbdd6b1aa, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.464536 2014 consensus_queue.cc:260] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.464593 2016 consensus_queue.cc:260] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.464628 2014 ts_tablet_manager.cc:1428] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:33.464691 2016 ts_tablet_manager.cc:1428] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:33.464712 2024 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.464802 2017 tablet_bootstrap.cc:654] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:33.464799 2024 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.465150 2018 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.465236 2018 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.465637 2017 tablet_bootstrap.cc:492] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: No bootstrap required, opened a new log
I20260925 14:50:33.465692 2017 ts_tablet_manager.cc:1397] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:33.465834 2017 raft_consensus.cc:348] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.465893 2017 raft_consensus.cc:374] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:33.465907 2017 raft_consensus.cc:729] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e64aed07c9434aff88ddaabd9e546c5e, State: Initialized, Role: FOLLOWER
I20260925 14:50:33.465971 2017 consensus_queue.cc:260] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.466055 2017 ts_tablet_manager.cc:1428] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:33.466142 2022 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.466249 2022 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.763589 2022 raft_consensus.cc:482] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:33.763727 2022 raft_consensus.cc:504] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.764107 2022 leader_election.cc:288] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649), c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3:44955)
I20260925 14:50:33.768543 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.764485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52397 (local address 127.0.98.2:44649)
0925 14:50:33.764628 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.764630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.764681 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:50:33.764779 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.764780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.764818 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.764900 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:33.765237 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.765980 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.767589 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.768192 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.768248 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.768257 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.768428 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.768470 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":54,"threads_started":1}
I20260925 14:50:33.768553 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.764373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44649 (local address 127.0.98.1:52397)
0925 14:50:33.764582 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.764611 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.764699 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.764942 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.764943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.764946 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:33.765139 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.765142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.766061 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.766094 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.767483 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.767486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.768287 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.768288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.768348 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.768352 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.768421 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.768464 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":103,"thread_start_us":103,"threads_started":1}
I20260925 14:50:33.768816 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.764616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44955 (local address 127.0.98.1:36365)
0925 14:50:33.764704 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.764714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.764772 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.764996 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.764997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.764999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:33.765130 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.765133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.766098 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.766098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.767454 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.767457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.768242 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.768243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.768351 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.768357 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.768560 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.768607 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":35,"thread_start_us":38,"threads_started":1}
I20260925 14:50:33.768944 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c1838134564ba98e4ff66671db2e8f" candidate_uuid: "e64aed07c9434aff88ddaabd9e546c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" is_pre_election: true
I20260925 14:50:33.769003 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.764693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36365 (local address 127.0.98.3:44955)
0925 14:50:33.764818 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.764820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.764828 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:33.764859 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.764859 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.764871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.764946 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:33.765197 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.765980 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.767548 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.768127 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.768201 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.768215 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.768496 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.768936 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 14:50:33.769045 1805 raft_consensus.cc:2454] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64aed07c9434aff88ddaabd9e546c5e in term 0.
I20260925 14:50:33.769219 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c1838134564ba98e4ff66671db2e8f" candidate_uuid: "e64aed07c9434aff88ddaabd9e546c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80c5a7226c44498bca5925dd25c7c40" is_pre_election: true
I20260925 14:50:33.769318 1945 raft_consensus.cc:2454] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e64aed07c9434aff88ddaabd9e546c5e in term 0.
I20260925 14:50:33.769276 1598 leader_election.cc:302] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5cc5d2c2b9c4475936b756bbdd6b1aa, e64aed07c9434aff88ddaabd9e546c5e; no voters:
I20260925 14:50:33.769455 2022 raft_consensus.cc:2781] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:33.769503 2022 raft_consensus.cc:482] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:33.769531 2022 raft_consensus.cc:3037] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.770200 2022 raft_consensus.cc:504] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.770304 2022 leader_election.cc:288] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [CANDIDATE]: Term 1 election: Requested vote from peers e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649), c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3:44955)
I20260925 14:50:33.770521 1805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c1838134564ba98e4ff66671db2e8f" candidate_uuid: "e64aed07c9434aff88ddaabd9e546c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa"
I20260925 14:50:33.770520 1945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c1838134564ba98e4ff66671db2e8f" candidate_uuid: "e64aed07c9434aff88ddaabd9e546c5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c80c5a7226c44498bca5925dd25c7c40"
I20260925 14:50:33.770594 1805 raft_consensus.cc:3037] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.770597 1945 raft_consensus.cc:3037] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:33.771189 1805 raft_consensus.cc:2454] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64aed07c9434aff88ddaabd9e546c5e in term 1.
I20260925 14:50:33.771220 1945 raft_consensus.cc:2454] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e64aed07c9434aff88ddaabd9e546c5e in term 1.
I20260925 14:50:33.771405 1598 leader_election.cc:302] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5cc5d2c2b9c4475936b756bbdd6b1aa, e64aed07c9434aff88ddaabd9e546c5e; no voters:
I20260925 14:50:33.771523 2022 raft_consensus.cc:2781] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:33.771665 2022 raft_consensus.cc:686] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [term 1 LEADER]: Becoming Leader. State: Replica: e64aed07c9434aff88ddaabd9e546c5e, State: Running, Role: LEADER
I20260925 14:50:33.771733 2022 consensus_queue.cc:237] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } }
I20260925 14:50:33.772051 2056 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: TxnStatusTablet state changed. Reason: New leader e64aed07c9434aff88ddaabd9e546c5e. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.772127 2056 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:33.772363 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:50:33.772481 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:50:33.772410 1507 catalog_manager.cc:5495] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e reported cstate change: term changed from 0 to 1, leader changed from <none> to e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:33.803272 2024 consensus_queue.cc:1035] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.804946 2062 consensus_queue.cc:1035] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.818470 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.815834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.98.1:41177)
0925 14:50:33.815988 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.815990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.815999 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:33.816032 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.816032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.816051 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.816140 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:33.816259 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.816770 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.817568 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.817804 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.817899 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.818077 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.818199 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.818283 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.818360 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.818381 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260925 14:50:33.818490 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.815758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41177 (local address 127.0.0.1:57510)
0925 14:50:33.815837 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.815850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.815903 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.816148 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.816149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.816150 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:33.816203 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.816206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.816856 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.816857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.817477 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.817479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.817889 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.817890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.817948 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.817969 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.818268 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.818278 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.818334 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.818412 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:50:33.819782 1805 raft_consensus.cc:1261] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Refusing update from remote peer e64aed07c9434aff88ddaabd9e546c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:33.819834 1945 raft_consensus.cc:1261] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Refusing update from remote peer e64aed07c9434aff88ddaabd9e546c5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:33.820077 2022 consensus_queue.cc:1035] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.820206 2056 consensus_queue.cc:1035] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:33.820832 2018 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: TxnStatusTablet state changed. Reason: New leader e64aed07c9434aff88ddaabd9e546c5e. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.820948 2018 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.821229 2062 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: TxnStatusTablet state changed. Reason: New leader e64aed07c9434aff88ddaabd9e546c5e. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.821316 2062 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.821909 2022 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.822000 2022 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:33.822069 2063 mvcc.cc:205] Tried to move back new op lower bound from 7333264727323734016 to 7333264727129546752. Current Snapshot: MvccSnapshot[applied={T|T < 7333264727323734016}]
I20260925 14:50:33.822077 2070 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.822172 2065 mvcc.cc:205] Tried to move back new op lower bound from 7333264727323734016 to 7333264727129546752. Current Snapshot: MvccSnapshot[applied={T|T < 7333264727323734016}]
I20260925 14:50:33.822420 2070 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e64aed07c9434aff88ddaabd9e546c5e: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:33.822402 2018 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.822496 2018 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P e5cc5d2c2b9c4475936b756bbdd6b1aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.822774 2061 mvcc.cc:205] Tried to move back new op lower bound from 7333264727323734016 to 7333264727129546752. Current Snapshot: MvccSnapshot[applied={T|T < 7333264727323734016}]
I20260925 14:50:33.822798 2062 tablet_replica.cc:439] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e64aed07c9434aff88ddaabd9e546c5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e64aed07c9434aff88ddaabd9e546c5e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41177 } } peers { permanent_uuid: "e5cc5d2c2b9c4475936b756bbdd6b1aa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44649 } } peers { permanent_uuid: "c80c5a7226c44498bca5925dd25c7c40" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44955 } } }
I20260925 14:50:33.822862 2062 tablet_replica.cc:442] T c9c1838134564ba98e4ff66671db2e8f P c80c5a7226c44498bca5925dd25c7c40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:33.829073 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.826429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41763 (local address 127.0.98.1:41177)
0925 14:50:33.826477 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.826479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.826661 (+ 182us) server_negotiation.cc:407] Connection header received
0925 14:50:33.826728 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.826729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.826737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.826783 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:33.826905 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.827334 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.828416 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.828612 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.828672 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.828855 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.828886 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.828924 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.829010 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.829027 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:50:33.829159 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.826364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41177 (local address 127.0.98.3:41763)
0925 14:50:33.826579 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.826591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.826659 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.826787 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.826788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.826790 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:33.826840 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.826841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.827403 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.827404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.828313 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.828316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.828692 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.828692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.828753 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.828769 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.828944 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.828946 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.829006 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.829055 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":134,"threads_started":1}
I20260925 14:50:33.841799 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.838784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49001 (local address 127.0.98.3:44955)
0925 14:50:33.838829 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.838831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.838943 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:50:33.838990 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.838990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.839003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.839078 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:33.839172 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.839976 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.841111 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.841293 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.841349 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.841572 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.841601 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.841652 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.841727 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.841746 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:33.841846 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.838716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44955 (local address 127.0.98.1:49001)
0925 14:50:33.838877 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:33.838888 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:33.838945 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:33.839078 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:33.839078 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:33.839081 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:33.839129 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.839130 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.840089 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.840090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.841017 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:33.841020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.841383 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.841384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:33.841454 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.841470 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:33.841678 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:33.841681 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:33.841727 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:50:33.841773 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260925 14:50:33.851450 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:33.851560 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:33.852183 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c28329db9584e9e81f4d9f34b4c1f21: Sending DeleteTablet for 3 replicas of tablet 2a825cf786d448b88d546ca3d68d2e94
I20260925 14:50:33.852443 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36348
I20260925 14:50:33.852447 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57498
I20260925 14:50:33.852661 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a825cf786d448b88d546ca3d68d2e94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41528
I20260925 14:50:33.852685 2080 tablet_replica.cc:330] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: stopping tablet replica
I20260925 14:50:33.852736 2080 raft_consensus.cc:2229] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:33.852792 2080 raft_consensus.cc:2258] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.853009 2082 tablet_replica.cc:330] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: stopping tablet replica
I20260925 14:50:33.853014 2080 ts_tablet_manager.cc:1905] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.853072 2082 raft_consensus.cc:2229] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:33.853128 2082 raft_consensus.cc:2258] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.853606 2082 ts_tablet_manager.cc:1905] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.854444 2081 tablet_replica.cc:330] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: stopping tablet replica
I20260925 14:50:33.854547 2081 raft_consensus.cc:2229] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:33.854628 2081 raft_consensus.cc:2258] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:33.854882 2081 ts_tablet_manager.cc:1905] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:33.855031 2080 ts_tablet_manager.cc:1918] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:33.855082 2080 log.cc:1196] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/2a825cf786d448b88d546ca3d68d2e94
I20260925 14:50:33.855311 2080 ts_tablet_manager.cc:1939] T 2a825cf786d448b88d546ca3d68d2e94 P e64aed07c9434aff88ddaabd9e546c5e: Deleting consensus metadata
I20260925 14:50:33.855422 2082 ts_tablet_manager.cc:1918] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:33.855473 2082 log.cc:1196] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/2a825cf786d448b88d546ca3d68d2e94
I20260925 14:50:33.855613 1495 catalog_manager.cc:4841] TS e64aed07c9434aff88ddaabd9e546c5e (127.0.98.1:41177): tablet 2a825cf786d448b88d546ca3d68d2e94 (table test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]) successfully deleted
I20260925 14:50:33.855715 2082 ts_tablet_manager.cc:1939] T 2a825cf786d448b88d546ca3d68d2e94 P c80c5a7226c44498bca5925dd25c7c40: Deleting consensus metadata
I20260925 14:50:33.856019 1494 catalog_manager.cc:4841] TS c80c5a7226c44498bca5925dd25c7c40 (127.0.98.3:44955): tablet 2a825cf786d448b88d546ca3d68d2e94 (table test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]) successfully deleted
I20260925 14:50:33.856473 2081 ts_tablet_manager.cc:1918] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:33.856513 2081 log.cc:1196] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/2a825cf786d448b88d546ca3d68d2e94
I20260925 14:50:33.856664 2081 ts_tablet_manager.cc:1939] T 2a825cf786d448b88d546ca3d68d2e94 P e5cc5d2c2b9c4475936b756bbdd6b1aa: Deleting consensus metadata
I20260925 14:50:33.857102 1494 catalog_manager.cc:4841] TS e5cc5d2c2b9c4475936b756bbdd6b1aa (127.0.98.2:44649): tablet 2a825cf786d448b88d546ca3d68d2e94 (table test-table [id=b5bdcc7407ed4657a5fd2fb2d51c03b8]) successfully deleted
Sep 25 14:50:33 dist-test-slave-gpc1 krb5kdc[1463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347832, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:33.864897 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.853832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.98.1:41177)
0925 14:50:33.853873 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.853875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.856911 (+ 3036us) server_negotiation.cc:407] Connection header received
0925 14:50:33.857011 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.857011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.857022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.857070 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:33.857724 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.858202 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.858959 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.859160 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.859232 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.861508 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:33.861516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:33.861517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:33.861524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:33.863655 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.864035 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.864038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.864039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.864135 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.864325 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.864326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.864327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.864502 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:33.864667 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.864709 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.864794 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:50:33.865201 1705 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57516
I20260925 14:50:33.872121 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:33.866239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.98.62:43393)
0925 14:50:33.866289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:33.866291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:33.866301 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:33.866387 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:33.866388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:33.866400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:33.866458 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:33.866939 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.867429 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.868107 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:33.868315 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:33.868398 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:33.869555 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:33.869557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:33.869557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:33.869564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:33.871020 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.871357 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.871358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.871358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.871412 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:33.871631 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:33.871632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:33.871632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:33.871744 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:33.871845 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:33.871944 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:50:33.872014 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 14:50:33.872390 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49968
I20260925 14:50:33.872859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1580
I20260925 14:50:33.878031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1720
I20260925 14:50:33.882788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1860
I20260925 14:50:33.888303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 1476
2026-09-25T14:50:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3147 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:33 dist-test-slave-gpc1 krb5kdc[2089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:33 dist-test-slave-gpc1 krb5kdc[2089](info): set up 2 sockets
Sep 25 14:50:33 dist-test-slave-gpc1 krb5kdc[2089](info): commencing operation
krb5kdc: starting...
W20260925 14:50:34.918074 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:35.035495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34611
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:35.116158 2102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:35.116364 2102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:35.116393 2102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:35.117528 2102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:35.117571 2102 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:50:35.117585 2102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:35.117599 2102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:35.117610 2102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:35.117622 2102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:35.119134 2102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:35.119395 2102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:35.119642 2102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:35.124534 2107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.124614 2102 server_base.cc:1034] running on GCE node
W20260925 14:50:35.124563 2108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.124574 2110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.124928 2102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:35.125180 2102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:35.126327 2102 hybrid_clock.cc:648] HybridClock initialized: now 1790347835126313 us; error 29 us; skew 500 ppm
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:35.128753 2102 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:35.129261 2102 webserver.cc:460] Webserver started at http://127.0.98.62:40625/ using document root <none> and password file <none>
I20260925 14:50:35.129492 2102 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:35.129561 2102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:35.129711 2102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:35.130859 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "05ad4a47eed84cde85dac56ad961578b"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "783119818b01215867b5d0593d0e1286"
server_key_iv: "2b027aa43a4d8a74c225e880ba56bfd4"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.131217 2102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "05ad4a47eed84cde85dac56ad961578b"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "783119818b01215867b5d0593d0e1286"
server_key_iv: "2b027aa43a4d8a74c225e880ba56bfd4"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.132536 2102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:50:35.133229 2117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.133396 2102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.133509 2102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "05ad4a47eed84cde85dac56ad961578b"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "783119818b01215867b5d0593d0e1286"
server_key_iv: "2b027aa43a4d8a74c225e880ba56bfd4"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.133582 2102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:35.143421 2102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:35.145162 2102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:35.145321 2102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:35.149981 2102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34611
I20260925 14:50:35.150049 2179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34611 every 8 connection(s)
I20260925 14:50:35.150447 2102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:35.151119 2180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.153379 2180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Bootstrap starting.
I20260925 14:50:35.153967 2180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.154255 2180 log.cc:824] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:35.154975 2180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: No bootstrap required, opened a new log
I20260925 14:50:35.157047 2180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } }
I20260925 14:50:35.157184 2180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.157207 2180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ad4a47eed84cde85dac56ad961578b, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.157382 2180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } }
I20260925 14:50:35.157449 2180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:35.157470 2180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:35.157498 2180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:35.158056 2180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } }
I20260925 14:50:35.158140 2180 leader_election.cc:302] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ad4a47eed84cde85dac56ad961578b; no voters:
I20260925 14:50:35.158378 2180 leader_election.cc:288] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:35.158427 2183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:35.158602 2183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [term 1 LEADER]: Becoming Leader. State: Replica: 05ad4a47eed84cde85dac56ad961578b, State: Running, Role: LEADER
I20260925 14:50:35.158680 2180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:35.158766 2183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } }
I20260925 14:50:35.159175 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ad4a47eed84cde85dac56ad961578b. Latest consensus state: current_term: 1 leader_uuid: "05ad4a47eed84cde85dac56ad961578b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } } }
I20260925 14:50:35.159276 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:35.159458 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ad4a47eed84cde85dac56ad961578b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ad4a47eed84cde85dac56ad961578b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34611 } } }
I20260925 14:50:35.159492 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:35.160169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2102
I20260925 14:50:35.160255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:35.160313 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:35.160436 392 external_mini_cluster.cc:1428] Setting key 521b33aba12b0b724d9ffa73172438ac
I20260925 14:50:35.160454 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:35.162377 2192 catalog_manager.cc:1347] Generated new cluster ID: 6ba70d72f3c34c48bf6b42385236783c
I20260925 14:50:35.162431 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.171430 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.160893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.98.62:34611)
0925 14:50:35.161413 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.161423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.161456 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:50:35.161577 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.161584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.161653 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.161804 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.162752 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.163771 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.164603 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.165069 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.165204 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.167781 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.167792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.167793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.167804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.169674 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.170088 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.170092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.170092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.170201 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.170470 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.170471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.170472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.170727 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.170931 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.171034 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.171204 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":82,"threads_started":1}
I20260925 14:50:35.172549 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:35.173098 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:35.180202 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Generated new TSK 0
I20260925 14:50:35.180399 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:35.208874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34611
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:35.290179 2206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:35.290433 2206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:35.290462 2206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:35.291613 2206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:35.291692 2206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:35.291716 2206 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:50:35.291744 2206 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:35.291764 2206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:35.293285 2206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:35.293547 2206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:35.293778 2206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:35.298880 2212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.298910 2211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.298902 2214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.299206 2206 server_base.cc:1034] running on GCE node
I20260925 14:50:35.299448 2206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:35.299652 2206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:35.300820 2206 hybrid_clock.cc:648] HybridClock initialized: now 1790347835300794 us; error 36 us; skew 500 ppm
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:35.303705 2206 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:35.304212 2206 webserver.cc:460] Webserver started at http://127.0.98.1:44629/ using document root <none> and password file <none>
I20260925 14:50:35.304437 2206 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:35.304504 2206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:35.304664 2206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:35.305732 2206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "0cb98b792fdd452d98b764537e472963"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "637243c2c0eae365de21000569934914"
server_key_iv: "17ee380fe69b901fb5a8c7844963fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.306104 2206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0cb98b792fdd452d98b764537e472963"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "637243c2c0eae365de21000569934914"
server_key_iv: "17ee380fe69b901fb5a8c7844963fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.307474 2206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:35.308182 2221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.308362 2206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:35.308452 2206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "0cb98b792fdd452d98b764537e472963"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "637243c2c0eae365de21000569934914"
server_key_iv: "17ee380fe69b901fb5a8c7844963fd2d"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.308537 2206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:35.316231 2206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:35.318014 2206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:35.318181 2206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:35.318806 2206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:35.318873 2206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.318915 2206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:35.318943 2206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.326082 2206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41329
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.326138 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41329 every 8 connection(s)
I20260925 14:50:35.326506 2206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:35.329276 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.319300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44557 (local address 127.0.98.62:34611)
0925 14:50:35.319373 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.319376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.319628 (+ 252us) server_negotiation.cc:407] Connection header received
0925 14:50:35.320449 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.320451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.320462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.320522 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.321454 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.322308 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.323419 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.323648 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.323743 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.326738 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.326739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.326740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.326747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.327984 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.328405 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.328406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.328406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.328456 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.328721 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.328722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.328722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.328824 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.328872 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.329148 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.329215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:50:35.329432 2230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.319275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.1:44557)
0925 14:50:35.319472 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.319527 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.320318 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.320607 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.320612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.321015 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.321344 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.321351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.322462 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.322463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.323293 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.323296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.323768 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.323769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.323965 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.324962 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.324971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.326611 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.328081 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.328088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.328090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.328322 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.328554 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.328554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.328555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.328659 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.328880 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.328884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.328999 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.329081 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.329217 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
I20260925 14:50:35.330123 2337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34611
I20260925 14:50:35.330196 2337 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:35.330420 2337 heartbeater.cc:511] Master 127.0.98.62:34611 requested a full tablet report, sending...
I20260925 14:50:35.330996 2134 ts_manager.cc:194] Registered new tserver with Master: 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329)
I20260925 14:50:35.331702 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44557
I20260925 14:50:35.334153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2206
I20260925 14:50:35.334264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:35.334481 392 external_mini_cluster.cc:1428] Setting key 495869e8eac0c94ff40b2a2f43b9633e
I20260925 14:50:35.337023 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.331295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45791 (local address 127.0.98.62:34611)
0925 14:50:35.331332 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.331334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.331465 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:50:35.331547 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.331547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.331556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.331600 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.331948 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.332458 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.333049 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.333241 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.333290 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.334107 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.334109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.334110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.334116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.335876 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.336282 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.336283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.336283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.336345 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.336582 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.336583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.336584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.336703 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.336755 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.336909 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.336966 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:50:35.337132 2343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.331226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.1:45791)
0925 14:50:35.331393 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.331402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.331490 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.331603 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.331604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.331809 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.331869 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.331871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.332525 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.332525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.332985 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.332986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.333302 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.333303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.333353 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.333746 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.333750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.334021 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.335952 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.335953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.335953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.336172 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.336417 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.336418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.336418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.336500 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.336766 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.336767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.336813 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.336893 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.336999 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:35.359061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34611
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:35.438782 2346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:35.438988 2346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:35.439013 2346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:35.440150 2346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:35.440192 2346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:35.440213 2346 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:50:35.440261 2346 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:35.440289 2346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:35.441709 2346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:35.441957 2346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:35.442183 2346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:35.447091 2351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.447148 2346 server_base.cc:1034] running on GCE node
W20260925 14:50:35.447129 2352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.447109 2354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.447482 2346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:35.447726 2346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:35.448884 2346 hybrid_clock.cc:648] HybridClock initialized: now 1790347835448874 us; error 37 us; skew 500 ppm
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:35.451238 2346 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:35.451704 2346 webserver.cc:460] Webserver started at http://127.0.98.2:39283/ using document root <none> and password file <none>
I20260925 14:50:35.451933 2346 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:35.452005 2346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:35.452152 2346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:35.453161 2346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "a06b64eb2c024ce78245d4d4d0ec68ac"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "2c9969842dfeda357e2187ca06ff0d33"
server_key_iv: "c96dce0927fc59ab69398e965f23ecdb"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.453519 2346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a06b64eb2c024ce78245d4d4d0ec68ac"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "2c9969842dfeda357e2187ca06ff0d33"
server_key_iv: "c96dce0927fc59ab69398e965f23ecdb"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.454844 2346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:50:35.455565 2361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.455724 2346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:35.455812 2346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "a06b64eb2c024ce78245d4d4d0ec68ac"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "2c9969842dfeda357e2187ca06ff0d33"
server_key_iv: "c96dce0927fc59ab69398e965f23ecdb"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.455892 2346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:35.483820 2346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:35.485725 2346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:35.485888 2346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:35.486515 2346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:35.486565 2346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.486595 2346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:35.486617 2346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.493846 2346 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42611
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.493992 2476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42611 every 8 connection(s)
I20260925 14:50:35.494256 2346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:35.497102 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.486929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51127 (local address 127.0.98.62:34611)
0925 14:50:35.487003 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.487006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.487319 (+ 313us) server_negotiation.cc:407] Connection header received
0925 14:50:35.488095 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.488097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.488109 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.488169 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.489114 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.489627 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.491315 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.491549 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.491642 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.494606 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.494608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.494609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.494618 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.495966 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.496330 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.496331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.496331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.496380 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.496586 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.496587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.496587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.496683 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.496731 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.496985 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.497041 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:50:35.497263 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.486920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.2:51127)
0925 14:50:35.487174 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.487232 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.487961 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.488264 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.488269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.488671 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.488989 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.488997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.489736 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.489737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.491217 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.491220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.491659 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.491660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.491780 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.492663 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.492670 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.494493 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.496040 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.496045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.496047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.496263 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.496436 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.496437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.496437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.496533 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.496734 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.496738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.496845 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.496922 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.497055 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":143,"threads_started":1}
I20260925 14:50:35.497756 2477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34611
I20260925 14:50:35.497848 2477 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:35.498049 2477 heartbeater.cc:511] Master 127.0.98.62:34611 requested a full tablet report, sending...
I20260925 14:50:35.498476 2134 ts_manager.cc:194] Registered new tserver with Master: a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611)
I20260925 14:50:35.498970 2134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51127
I20260925 14:50:35.504117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2346
I20260925 14:50:35.504230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:35.504228 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.498827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33961 (local address 127.0.98.62:34611)
0925 14:50:35.498872 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.498873 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.498973 (+ 100us) server_negotiation.cc:407] Connection header received
0925 14:50:35.499054 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.499055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.499065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.499122 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.499424 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.499882 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.500583 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.500788 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.500869 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.501836 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.501838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.501838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.501845 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.503091 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.503427 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.503428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.503429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.503479 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.503688 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.503689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.503690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.503832 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.503906 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.504065 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.504138 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:35.504243 2483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.498740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.2:33961)
0925 14:50:35.498910 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.498921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.498993 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.499129 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.499130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.499298 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.499347 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.499348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.499988 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.499989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.500502 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.500503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.500872 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.500873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.500946 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.501395 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.501400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.501768 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.503172 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.503173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.503173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.503342 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.503553 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.503554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.503554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.503610 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.503912 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.503913 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.503954 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.504041 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.504140 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260925 14:50:35.504442 392 external_mini_cluster.cc:1428] Setting key 06b343ae07d4f01f540bade02cd52719
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:35.528574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34611
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:35.608270 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:35.608520 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:35.608560 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:35.609753 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:35.609828 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:35.609863 2486 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:50:35.609915 2486 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:50:35.609946 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:35.611418 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:35.611693 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:35.611931 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:35.616879 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.616978 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:35.616880 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:35.617125 2486 server_base.cc:1034] running on GCE node
I20260925 14:50:35.617383 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:35.617622 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:35.618781 2486 hybrid_clock.cc:648] HybridClock initialized: now 1790347835618769 us; error 24 us; skew 500 ppm
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:35.621073 2486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:35.621501 2486 webserver.cc:460] Webserver started at http://127.0.98.3:44335/ using document root <none> and password file <none>
I20260925 14:50:35.621724 2486 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:35.621789 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:35.621939 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:35.623023 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "87728b96f2b747ce9e82f758c0b15190"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "68546ef85dfff23f5b3e54f57daa9214"
server_key_iv: "cd66e495e910895e4e6dfca47eafe6ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.623383 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87728b96f2b747ce9e82f758c0b15190"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "68546ef85dfff23f5b3e54f57daa9214"
server_key_iv: "cd66e495e910895e4e6dfca47eafe6ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.624711 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:35.625452 2501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.625630 2486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:35.625728 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "87728b96f2b747ce9e82f758c0b15190"
format_stamp: "Formatted at 2026-09-25 14:50:35 on dist-test-slave-gpc1"
server_key: "68546ef85dfff23f5b3e54f57daa9214"
server_key_iv: "cd66e495e910895e4e6dfca47eafe6ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:35.625805 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:35.645627 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:35.647607 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:35.647775 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:35.648285 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:35.648350 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.648393 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:35.648451 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.655639 2486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37935
I20260925 14:50:35.655827 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37935 every 8 connection(s)
I20260925 14:50:35.656032 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:35.658152 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.648767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51473 (local address 127.0.98.62:34611)
0925 14:50:35.648835 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.648837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.649121 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:50:35.649653 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.649655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.649668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.649717 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.650561 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.651069 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.652163 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.652370 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.652477 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.655152 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.655153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.655154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.655161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.656740 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.657201 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.657202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.657203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.657264 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.657520 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.657521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.657521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.657635 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.657703 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.657981 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.658064 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:50:35.658373 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.648754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.3:51473)
0925 14:50:35.648972 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.649012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.649542 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.649785 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.649789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.650171 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.650469 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.650474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.651178 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.651179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.652080 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.652082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.652485 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.652486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.652615 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.653160 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.653170 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.655065 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.656835 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.656841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.656843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.657113 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.657330 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.657330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.657331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.657444 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.657705 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.657711 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.657841 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.657938 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.658111 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260925 14:50:35.658838 2617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34611
I20260925 14:50:35.658937 2617 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:35.659117 2617 heartbeater.cc:511] Master 127.0.98.62:34611 requested a full tablet report, sending...
I20260925 14:50:35.659502 2133 ts_manager.cc:194] Registered new tserver with Master: 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:35.660038 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51473
I20260925 14:50:35.663367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2486
I20260925 14:50:35.663484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:35.663690 392 external_mini_cluster.cc:1428] Setting key 427e44d277d5d81571147edf5780b83e
I20260925 14:50:35.664412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:35.665230 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.659901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56205 (local address 127.0.98.62:34611)
0925 14:50:35.659953 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.659954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.660146 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:50:35.660204 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.660205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.660215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.660274 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.660725 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.661163 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.661879 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.662080 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.662149 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.662955 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.662956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.662956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.662962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.664187 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.664521 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.664522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.664522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.664574 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.664749 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.664750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.664751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.664844 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.664914 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.665075 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.665144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:35.665246 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.659863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.98.3:56205)
0925 14:50:35.660074 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.660085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.660159 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.660279 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.660280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.660453 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.660499 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.660500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.661270 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.661271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.661799 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.661800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.662158 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.662159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.662230 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.662607 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.662611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.662856 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.664278 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.664279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.664279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.664443 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.664634 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.664636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.664636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.664690 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.664907 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.664908 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.664966 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.665052 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.665145 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":144,"threads_started":1}
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.685778 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.679264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.0.98.62:34611)
0925 14:50:35.679324 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.679326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.679335 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:35.679359 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.679360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.679376 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.679424 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.679945 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.680471 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.681193 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.681369 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.681422 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.683550 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.683551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.683552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.683558 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.684900 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.685200 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.685201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.685201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.685247 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.685400 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.685400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.685400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.685490 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.685560 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.685640 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.685692 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:50:35.687265 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:35.688757 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:35.695318 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.691226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.0.98.3:37935)
0925 14:50:35.691401 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.691405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.691722 (+ 317us) server_negotiation.cc:407] Connection header received
0925 14:50:35.691766 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.691768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.691784 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.691848 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:35.692041 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.693091 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694116 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.694854 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694935 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.694995 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.695163 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.695206 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260925 14:50:35.695313 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.690996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:55634)
0925 14:50:35.691140 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.691178 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.691240 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.691635 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.691640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.691642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:35.691710 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.691712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.693138 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.693138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.694307 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.694308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.695016 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.695016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.695074 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.695077 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.695111 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.695162 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":46,"threads_started":1}
I20260925 14:50:35.695313 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.691011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37935 (local address 127.0.0.1:54856)
0925 14:50:35.691620 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.691629 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.691668 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.691914 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.691915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.691917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:35.691951 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.691953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.693185 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.693186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.694034 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.694037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.694970 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.695049 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.695064 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.695136 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.695164 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":90,"threads_started":1}
I20260925 14:50:35.695653 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.691186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.98.1:41329)
0925 14:50:35.691373 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.691378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.691570 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:50:35.691615 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.691617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.691635 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.691768 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:35.691977 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.692683 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694159 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.694866 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694936 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.694995 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.695310 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.695356 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260925 14:50:35.695794 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.690996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41329 (local address 127.0.0.1:52768)
0925 14:50:35.691479 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.691488 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.691547 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.691835 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.691836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.691838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:35.691885 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.691886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.692768 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.692769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.694077 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.694079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.695044 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.695046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.695099 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.695102 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.695174 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.695212 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":32,"thread_start_us":74,"threads_started":1}
I20260925 14:50:35.696158 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.691275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.98.2:42611)
0925 14:50:35.691428 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.691432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.691441 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:35.691481 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.691483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.691503 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.691788 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:35.691846 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.693016 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.694429 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.694924 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.695779 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.695851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.695881 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.696083 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260925 14:50:35.696184 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 (DEFAULT_TABLE table=test-table [id=97eab996f4634112ae3d2d721d506af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:35.696445 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 (DEFAULT_TABLE table=test-table [id=97eab996f4634112ae3d2d721d506af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:35.696592 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90cdc7dbb83c47c490d74c9864f6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.696775 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90cdc7dbb83c47c490d74c9864f6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.696698 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 (DEFAULT_TABLE table=test-table [id=97eab996f4634112ae3d2d721d506af6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:35.696997 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90cdc7dbb83c47c490d74c9864f6dad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.699465 2640 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Bootstrap starting.
I20260925 14:50:35.700276 2640 tablet_bootstrap.cc:654] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.700450 2641 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Bootstrap starting.
I20260925 14:50:35.700670 2642 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Bootstrap starting.
I20260925 14:50:35.700769 2640 log.cc:824] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:35.701081 2641 tablet_bootstrap.cc:654] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.701331 2642 tablet_bootstrap.cc:654] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.701442 2641 log.cc:824] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:35.701628 2640 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: No bootstrap required, opened a new log
I20260925 14:50:35.701692 2640 ts_tablet_manager.cc:1397] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:35.701681 2642 log.cc:824] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:35.702212 2641 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: No bootstrap required, opened a new log
I20260925 14:50:35.702610 2641 ts_tablet_manager.cc:1397] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:35.702878 2642 tablet_bootstrap.cc:492] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: No bootstrap required, opened a new log
I20260925 14:50:35.702960 2642 ts_tablet_manager.cc:1397] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:35.703774 2640 raft_consensus.cc:348] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.703945 2640 raft_consensus.cc:374] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.703984 2640 raft_consensus.cc:729] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.704105 2640 consensus_queue.cc:260] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.704362 2640 ts_tablet_manager.cc:1428] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:35.704384 2617 heartbeater.cc:503] Master 127.0.98.62:34611 was elected leader, sending a full tablet report...
I20260925 14:50:35.704533 2642 raft_consensus.cc:348] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.704702 2642 raft_consensus.cc:374] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.704638 2641 raft_consensus.cc:348] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.704741 2642 raft_consensus.cc:729] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb98b792fdd452d98b764537e472963, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.704789 2641 raft_consensus.cc:374] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.704830 2641 raft_consensus.cc:729] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.704850 2642 consensus_queue.cc:260] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.704959 2641 consensus_queue.cc:260] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.705102 2642 ts_tablet_manager.cc:1428] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:50:35.705152 2337 heartbeater.cc:503] Master 127.0.98.62:34611 was elected leader, sending a full tablet report...
I20260925 14:50:35.705209 2641 ts_tablet_manager.cc:1428] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:50:35.705332 2477 heartbeater.cc:503] Master 127.0.98.62:34611 was elected leader, sending a full tablet report...
I20260925 14:50:35.711191 2648 raft_consensus.cc:482] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:35.711256 2648 raft_consensus.cc:504] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.711570 2648 leader_election.cc:288] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:35.715276 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.711753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48201 (local address 127.0.98.1:41329)
0925 14:50:35.711913 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.711915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.711924 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:35.711966 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.711967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.711978 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.712031 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:35.712197 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.712939 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.714387 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.714931 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.715006 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.715016 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.715174 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.715213 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260925 14:50:35.715281 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.711817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34803 (local address 127.0.98.3:37935)
0925 14:50:35.711874 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.711876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.711954 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:50:35.711984 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.711984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.711994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.712046 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:35.712225 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.712947 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.714387 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.714940 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.715006 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.715016 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.715174 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.715227 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:50:35.715314 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.711734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41329 (local address 127.0.98.2:48201)
0925 14:50:35.711791 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.711803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.711841 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.712044 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.712045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.712047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:35.712110 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.712112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.713024 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.713025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.714277 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.714280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.715015 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.715016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.715090 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.715096 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.715155 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.715208 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":61}
I20260925 14:50:35.715323 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.711734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37935 (local address 127.0.98.2:34803)
0925 14:50:35.711876 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.711886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.711949 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.712077 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.712078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.712080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:35.712132 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.712133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.713024 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.713025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.714277 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.714280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.715015 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.715016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.715090 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.715097 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.715155 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.715207 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":71,"thread_start_us":82,"threads_started":1}
I20260925 14:50:35.715646 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90cdc7dbb83c47c490d74c9864f6dad2" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190" is_pre_election: true
I20260925 14:50:35.715775 2571 raft_consensus.cc:2454] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:35.715981 2365 leader_election.cc:302] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87728b96f2b747ce9e82f758c0b15190, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:35.716151 2648 raft_consensus.cc:2781] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:35.716207 2648 raft_consensus.cc:482] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:35.716223 2648 raft_consensus.cc:3037] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:35.716177 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90cdc7dbb83c47c490d74c9864f6dad2" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963" is_pre_election: true
I20260925 14:50:35.716323 2291 raft_consensus.cc:2454] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:35.716969 2648 raft_consensus.cc:504] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.717105 2648 leader_election.cc:288] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Requested vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:35.717254 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90cdc7dbb83c47c490d74c9864f6dad2" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963"
I20260925 14:50:35.717330 2291 raft_consensus.cc:3037] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:35.718046 2291 raft_consensus.cc:2454] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:35.718163 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90cdc7dbb83c47c490d74c9864f6dad2" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190"
I20260925 14:50:35.718245 2571 raft_consensus.cc:3037] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:35.718603 2362 leader_election.cc:302] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:35.718716 2648 raft_consensus.cc:2781] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:35.718901 2648 raft_consensus.cc:686] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Becoming Leader. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Running, Role: LEADER
I20260925 14:50:35.718995 2648 consensus_queue.cc:237] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.719182 2571 raft_consensus.cc:2454] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:35.719803 2134 catalog_manager.cc:5495] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } health_report { overall_health: UNKNOWN } } }
W20260925 14:50:35.744874 2478 tablet.cc:2369] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:50:35 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:35.757661 2202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.751134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.98.62:34611)
0925 14:50:35.751216 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:35.751218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:35.751292 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:50:35.751339 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:35.751340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:35.751349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:35.751400 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:35.751865 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.752343 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.753019 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.753191 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.753244 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.755293 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:35.755294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:35.755294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:35.755307 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:35.756468 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.756838 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.756839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.756840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.756892 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:35.757108 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:35.757109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:35.757109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:35.757212 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:35.757286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:35.757442 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:50:35.757546 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:50:35.757730 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:35.751051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34611 (local address 127.0.0.1:49228)
0925 14:50:35.751216 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:35.751226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:35.751295 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:35.751442 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:35.751443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:35.751753 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:35.751802 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.751804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.752434 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.752435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.752948 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:35.752950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:35.753254 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:35.753255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:35.753294 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:35.753795 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:35.753799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:35.755188 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:35.756562 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.756568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.756571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.756754 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.756978 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:35.756979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:35.756979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:35.757038 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:35.757276 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:35.757283 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:35.757326 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:50:35.757409 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:50:35.757604 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":76,"thread_start_us":79,"threads_started":1}
I20260925 14:50:35.760061 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49228:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:50:35.760304 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:35.761708 2551 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b63e50b7547c6a4913e404a14fbf5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cf386a1e83474f86dd1cef322a23f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:35.761739 2411 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b63e50b7547c6a4913e404a14fbf5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cf386a1e83474f86dd1cef322a23f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:35.761737 2271 tablet_service.cc:1467] Processing CreateTablet for tablet fc2b63e50b7547c6a4913e404a14fbf5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cf386a1e83474f86dd1cef322a23f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:50:35.761879 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2b63e50b7547c6a4913e404a14fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.761880 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2b63e50b7547c6a4913e404a14fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.761910 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2b63e50b7547c6a4913e404a14fbf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:35.763159 2642 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: Bootstrap starting.
I20260925 14:50:35.763365 2640 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: Bootstrap starting.
I20260925 14:50:35.763913 2642 tablet_bootstrap.cc:654] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.763926 2640 tablet_bootstrap.cc:654] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.764307 2641 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: Bootstrap starting.
I20260925 14:50:35.764657 2640 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: No bootstrap required, opened a new log
I20260925 14:50:35.764703 2640 ts_tablet_manager.cc:1397] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:35.764739 2641 tablet_bootstrap.cc:654] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:35.764853 2640 raft_consensus.cc:348] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.764943 2640 raft_consensus.cc:374] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.764968 2640 raft_consensus.cc:729] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.765053 2640 consensus_queue.cc:260] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.765188 2640 ts_tablet_manager.cc:1428] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:35.765324 2646 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:35.765416 2646 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:35.765509 2641 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: No bootstrap required, opened a new log
I20260925 14:50:35.765551 2641 ts_tablet_manager.cc:1397] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:35.765650 2642 tablet_bootstrap.cc:492] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: No bootstrap required, opened a new log
I20260925 14:50:35.765686 2642 ts_tablet_manager.cc:1397] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:50:35.765686 2641 raft_consensus.cc:348] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.765758 2641 raft_consensus.cc:374] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.765781 2641 raft_consensus.cc:729] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.765823 2641 consensus_queue.cc:260] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.765827 2642 raft_consensus.cc:348] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.765890 2642 raft_consensus.cc:374] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:35.765925 2642 raft_consensus.cc:729] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb98b792fdd452d98b764537e472963, State: Initialized, Role: FOLLOWER
I20260925 14:50:35.765918 2641 ts_tablet_manager.cc:1428] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.765970 2642 consensus_queue.cc:260] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:35.765969 2648 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:35.766067 2642 ts_tablet_manager.cc:1428] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:35.766068 2648 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:35.766160 2647 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:35.766245 2647 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:50:35.827246 2338 tablet.cc:2369] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:35.906652 2618 tablet.cc:2369] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:36.014808 2648 raft_consensus.cc:482] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:36.014910 2648 raft_consensus.cc:504] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.015113 2648 leader_election.cc:288] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:36.015374 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2b63e50b7547c6a4913e404a14fbf5" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963" is_pre_election: true
I20260925 14:50:36.015374 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2b63e50b7547c6a4913e404a14fbf5" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190" is_pre_election: true
I20260925 14:50:36.015483 2291 raft_consensus.cc:2454] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:36.015492 2571 raft_consensus.cc:2454] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:36.015632 2362 leader_election.cc:302] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:36.015728 2648 raft_consensus.cc:2781] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:36.015810 2648 raft_consensus.cc:482] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:36.015843 2648 raft_consensus.cc:3037] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.016523 2648 raft_consensus.cc:504] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.016669 2648 leader_election.cc:288] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Requested vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:36.016834 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2b63e50b7547c6a4913e404a14fbf5" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963"
I20260925 14:50:36.016914 2291 raft_consensus.cc:3037] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.016857 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2b63e50b7547c6a4913e404a14fbf5" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190"
I20260925 14:50:36.016955 2571 raft_consensus.cc:3037] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.017532 2291 raft_consensus.cc:2454] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:36.017649 2571 raft_consensus.cc:2454] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:36.017694 2362 leader_election.cc:302] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:36.017774 2648 raft_consensus.cc:2781] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:36.017838 2648 raft_consensus.cc:686] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Becoming Leader. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Running, Role: LEADER
I20260925 14:50:36.017889 2648 consensus_queue.cc:237] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.018169 2650 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: TxnStatusTablet state changed. Reason: New leader a06b64eb2c024ce78245d4d4d0ec68ac. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.018255 2650 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:36.018420 2662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:50:36.018543 2662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:50:36.018517 2133 catalog_manager.cc:5495] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:36.048736 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.046157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.0.1:55642)
0925 14:50:36.046236 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.046248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.046295 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.046473 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.046474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.046475 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:36.046527 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.046530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.047193 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.047195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.047772 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.047773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.048157 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.048158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.048232 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.048257 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.048549 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.048560 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.048619 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.048669 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:50:36.048725 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.046247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.0.98.2:42611)
0925 14:50:36.046294 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.046296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.046306 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:36.046391 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.046392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.046412 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.046470 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.046613 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.047105 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.047857 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.048082 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.048158 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.048346 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.048459 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.048571 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.048622 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.048645 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:36.050576 2571 raft_consensus.cc:1261] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.050653 2291 raft_consensus.cc:1261] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.050966 2650 consensus_queue.cc:1035] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.051081 2648 consensus_queue.cc:1035] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.052177 2646 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: TxnStatusTablet state changed. Reason: New leader a06b64eb2c024ce78245d4d4d0ec68ac. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.052228 2647 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: TxnStatusTablet state changed. Reason: New leader a06b64eb2c024ce78245d4d4d0ec68ac. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.052289 2646 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:36.052312 2647 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:36.053573 2650 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.053660 2650 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:36.053756 2647 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.053838 2647 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 0cb98b792fdd452d98b764537e472963: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:36.054246 2665 mvcc.cc:205] Tried to move back new op lower bound from 7333264736460627968 to 7333264736329695232. Current Snapshot: MvccSnapshot[applied={T|T < 7333264736460627968}]
I20260925 14:50:36.054337 2650 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.054431 2650 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P a06b64eb2c024ce78245d4d4d0ec68ac: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:50:36.054580 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333264736460627968 to 7333264736329695232. Current Snapshot: MvccSnapshot[applied={T|T < 7333264736460627968 or (T in {7333264736460627968})}]
I20260925 14:50:36.054896 2646 tablet_replica.cc:439] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } } }
I20260925 14:50:36.054979 2646 tablet_replica.cc:442] T fc2b63e50b7547c6a4913e404a14fbf5 P 87728b96f2b747ce9e82f758c0b15190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:50:36.058810 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.056106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.98.2:42611)
0925 14:50:36.056160 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.056161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.056173 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:50:36.056251 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.056252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.056259 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.056294 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.056429 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.056850 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.057975 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.058304 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.058390 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.058535 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.058602 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.058646 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.058723 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.058741 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:50:36.063848 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.061245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42853 (local address 127.0.98.2:42611)
0925 14:50:36.061283 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.061284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.061373 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:50:36.061434 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.061435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.061443 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.061489 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.061626 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.062059 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.063204 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.063368 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.063418 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.063646 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.063670 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.063700 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.063765 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.063787 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":21}
I20260925 14:50:36.063941 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.061155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.2:42853)
0925 14:50:36.061301 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.061311 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.061370 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.061495 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.061495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.061497 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:36.061573 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.061574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.062137 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.062138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.063130 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.063133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.063447 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.063448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.063537 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.063551 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.063715 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.063717 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.063764 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.063813 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260925 14:50:36.065975 2571 raft_consensus.cc:1261] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.066004 2291 raft_consensus.cc:1261] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.066200 2672 consensus_queue.cc:1035] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.066277 2650 consensus_queue.cc:1035] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.080250 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:36.080365 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:36.081092 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Sending DeleteTablet for 3 replicas of tablet 90cdc7dbb83c47c490d74c9864f6dad2
I20260925 14:50:36.081508 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260925 14:50:36.081609 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260925 14:50:36.081772 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 90cdc7dbb83c47c490d74c9864f6dad2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260925 14:50:36.081796 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:36.081938 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:36.082003 2681 tablet_replica.cc:330] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: stopping tablet replica
I20260925 14:50:36.082126 2681 raft_consensus.cc:2229] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:36.082216 2682 tablet_replica.cc:330] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: stopping tablet replica
I20260925 14:50:36.082275 2681 raft_consensus.cc:2258] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:36.082803 2683 tablet_replica.cc:330] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: stopping tablet replica
I20260925 14:50:36.082906 2683 raft_consensus.cc:2229] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:36.083221 2681 ts_tablet_manager.cc:1905] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:36.083271 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 41b891b96586407fb79fa9dd2297bc21 (DEFAULT_TABLE table=test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:36.083370 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b891b96586407fb79fa9dd2297bc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:36.083179 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 41b891b96586407fb79fa9dd2297bc21 (DEFAULT_TABLE table=test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:36.084666 2642 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Bootstrap starting.
I20260925 14:50:36.084753 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b891b96586407fb79fa9dd2297bc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:36.085106 2642 tablet_bootstrap.cc:654] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:36.085785 2642 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: No bootstrap required, opened a new log
I20260925 14:50:36.085837 2642 ts_tablet_manager.cc:1397] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:36.085970 2640 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Bootstrap starting.
I20260925 14:50:36.085973 2642 raft_consensus.cc:348] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.086030 2642 raft_consensus.cc:374] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:36.086030 2682 raft_consensus.cc:2229] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:36.086050 2642 raft_consensus.cc:729] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb98b792fdd452d98b764537e472963, State: Initialized, Role: FOLLOWER
I20260925 14:50:36.086087 2682 raft_consensus.cc:2258] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:36.086089 2642 consensus_queue.cc:260] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.086169 2642 ts_tablet_manager.cc:1428] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:36.083020 2410 tablet_service.cc:1467] Processing CreateTablet for tablet 41b891b96586407fb79fa9dd2297bc21 (DEFAULT_TABLE table=test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:36.086378 2640 tablet_bootstrap.cc:654] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:36.086367 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b891b96586407fb79fa9dd2297bc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:36.087721 2640 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: No bootstrap required, opened a new log
I20260925 14:50:36.087770 2640 ts_tablet_manager.cc:1397] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:50:36.087767 2641 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Bootstrap starting.
I20260925 14:50:36.087906 2640 raft_consensus.cc:348] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.087947 2683 raft_consensus.cc:2258] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:36.087981 2640 raft_consensus.cc:374] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:36.088003 2640 raft_consensus.cc:729] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Initialized, Role: FOLLOWER
I20260925 14:50:36.088044 2641 tablet_bootstrap.cc:654] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:36.088047 2640 consensus_queue.cc:260] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.088136 2640 ts_tablet_manager.cc:1428] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:36.088513 2683 ts_tablet_manager.cc:1905] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:36.089804 2683 ts_tablet_manager.cc:1918] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:36.089861 2683 log.cc:1196] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/90cdc7dbb83c47c490d74c9864f6dad2
I20260925 14:50:36.090041 2683 ts_tablet_manager.cc:1939] T 90cdc7dbb83c47c490d74c9864f6dad2 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting consensus metadata
I20260925 14:50:36.090273 2682 ts_tablet_manager.cc:1905] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:36.090601 2120 catalog_manager.cc:4841] TS a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611): tablet 90cdc7dbb83c47c490d74c9864f6dad2 (table test-table [id=97eab996f4634112ae3d2d721d506af6]) successfully deleted
I20260925 14:50:36.091535 2681 ts_tablet_manager.cc:1918] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:36.091626 2681 log.cc:1196] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/90cdc7dbb83c47c490d74c9864f6dad2
I20260925 14:50:36.091661 2682 ts_tablet_manager.cc:1918] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:36.091722 2682 log.cc:1196] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/90cdc7dbb83c47c490d74c9864f6dad2
I20260925 14:50:36.091912 2681 ts_tablet_manager.cc:1939] T 90cdc7dbb83c47c490d74c9864f6dad2 P 0cb98b792fdd452d98b764537e472963: Deleting consensus metadata
I20260925 14:50:36.091984 2682 ts_tablet_manager.cc:1939] T 90cdc7dbb83c47c490d74c9864f6dad2 P 87728b96f2b747ce9e82f758c0b15190: Deleting consensus metadata
I20260925 14:50:36.092351 2121 catalog_manager.cc:4841] TS 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935): tablet 90cdc7dbb83c47c490d74c9864f6dad2 (table test-table [id=97eab996f4634112ae3d2d721d506af6]) successfully deleted
I20260925 14:50:36.092377 2118 catalog_manager.cc:4841] TS 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329): tablet 90cdc7dbb83c47c490d74c9864f6dad2 (table test-table [id=97eab996f4634112ae3d2d721d506af6]) successfully deleted
I20260925 14:50:36.092545 2641 tablet_bootstrap.cc:492] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: No bootstrap required, opened a new log
I20260925 14:50:36.092592 2641 ts_tablet_manager.cc:1397] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:50:36.092736 2641 raft_consensus.cc:348] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.092803 2641 raft_consensus.cc:374] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:36.092823 2641 raft_consensus.cc:729] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Initialized, Role: FOLLOWER
I20260925 14:50:36.092864 2641 consensus_queue.cc:260] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.092937 2641 ts_tablet_manager.cc:1428] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:36.370066 2646 raft_consensus.cc:482] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:36.370167 2646 raft_consensus.cc:504] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.370591 2646 leader_election.cc:288] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611)
I20260925 14:50:36.374953 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.370984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60431 (local address 127.0.98.2:42611)
0925 14:50:36.371034 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.371036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.371044 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:36.371105 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.371106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.371117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.371185 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:36.371414 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.372169 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.373904 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.374482 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.374559 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.374574 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.374822 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.374878 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:50:36.374955 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.370879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58085 (local address 127.0.98.1:41329)
0925 14:50:36.371005 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.371008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.371021 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:50:36.371105 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.371106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.371117 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.371194 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:36.371428 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.372165 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.373904 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.374528 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.374595 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.374605 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.374823 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.374875 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260925 14:50:36.374969 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.370806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.3:60431)
0925 14:50:36.370955 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.370970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.371038 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.371202 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.371202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.371207 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:36.371323 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.371326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.372266 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.372267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.373794 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.373797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.374598 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.374600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.374700 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.374705 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.374778 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.374840 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":73,"thread_start_us":78,"threads_started":1}
I20260925 14:50:36.374974 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.370808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41329 (local address 127.0.98.3:58085)
0925 14:50:36.370973 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.370982 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.371038 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.371202 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.371202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.371204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:36.371324 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.371326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.372267 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.372267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.373794 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.373797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.374602 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.374603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.374700 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.374706 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.374778 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.374840 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":83,"thread_start_us":99,"threads_started":1}
I20260925 14:50:36.375258 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b891b96586407fb79fa9dd2297bc21" candidate_uuid: "87728b96f2b747ce9e82f758c0b15190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963" is_pre_election: true
I20260925 14:50:36.375349 2291 raft_consensus.cc:2454] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87728b96f2b747ce9e82f758c0b15190 in term 0.
I20260925 14:50:36.375465 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b891b96586407fb79fa9dd2297bc21" candidate_uuid: "87728b96f2b747ce9e82f758c0b15190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" is_pre_election: true
I20260925 14:50:36.375537 2502 leader_election.cc:302] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, 87728b96f2b747ce9e82f758c0b15190; no voters:
I20260925 14:50:36.375607 2431 raft_consensus.cc:2454] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87728b96f2b747ce9e82f758c0b15190 in term 0.
I20260925 14:50:36.375689 2646 raft_consensus.cc:2781] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:36.375730 2646 raft_consensus.cc:482] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:36.375754 2646 raft_consensus.cc:3037] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.376470 2646 raft_consensus.cc:504] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.376593 2646 leader_election.cc:288] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [CANDIDATE]: Term 1 election: Requested vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611)
I20260925 14:50:36.376784 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b891b96586407fb79fa9dd2297bc21" candidate_uuid: "87728b96f2b747ce9e82f758c0b15190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963"
I20260925 14:50:36.376806 2431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b891b96586407fb79fa9dd2297bc21" candidate_uuid: "87728b96f2b747ce9e82f758c0b15190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac"
I20260925 14:50:36.376880 2291 raft_consensus.cc:3037] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.376880 2431 raft_consensus.cc:3037] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:36.377674 2291 raft_consensus.cc:2454] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87728b96f2b747ce9e82f758c0b15190 in term 1.
I20260925 14:50:36.377811 2431 raft_consensus.cc:2454] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87728b96f2b747ce9e82f758c0b15190 in term 1.
I20260925 14:50:36.377846 2502 leader_election.cc:302] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, 87728b96f2b747ce9e82f758c0b15190; no voters:
I20260925 14:50:36.377980 2646 raft_consensus.cc:2781] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:36.378136 2646 raft_consensus.cc:686] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 1 LEADER]: Becoming Leader. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Running, Role: LEADER
I20260925 14:50:36.378212 2646 consensus_queue.cc:237] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:36.379005 2134 catalog_manager.cc:5495] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "87728b96f2b747ce9e82f758c0b15190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } health_report { overall_health: HEALTHY } } }
I20260925 14:50:36.444744 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.442173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.98.3:37935)
0925 14:50:36.442373 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.442375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.442382 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:50:36.442416 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.442416 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.442434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.442507 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.442642 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.443132 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.443938 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.444139 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.444188 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.444335 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.444452 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.444508 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.444608 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.444623 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260925 14:50:36.449427 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.447192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42611 (local address 127.0.98.3:50755)
0925 14:50:36.447340 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.447352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.447408 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.447530 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.447531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.447532 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:36.447576 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.447577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.448185 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.448186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.448744 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.448745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.449096 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.449096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.449145 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.449155 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.449298 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.449300 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.449333 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.449364 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 14:50:36.449438 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.447292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50755 (local address 127.0.98.2:42611)
0925 14:50:36.447349 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.447351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.447423 (+ 72us) server_negotiation.cc:407] Connection header received
0925 14:50:36.447446 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.447446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.447455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.447507 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.447617 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.448119 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.448812 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.449017 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.449077 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.449222 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.449247 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.449288 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.449335 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.449352 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:36.451684 2291 raft_consensus.cc:1261] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Refusing update from remote peer 87728b96f2b747ce9e82f758c0b15190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.451779 2431 raft_consensus.cc:1261] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Refusing update from remote peer 87728b96f2b747ce9e82f758c0b15190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:36.451912 2646 consensus_queue.cc:1035] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.452086 2690 consensus_queue.cc:1035] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:36.459210 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.456667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37935 (local address 127.0.98.2:45269)
0925 14:50:36.456746 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:36.456758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:36.456807 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:36.456943 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:36.456944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:36.456946 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:50:36.456985 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.456987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.457550 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.457551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.458265 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:36.458267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.458817 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.458817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:36.458877 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.458882 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.459031 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.459033 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:50:36.459081 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:50:36.459126 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 14:50:36.459244 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:36.456698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45269 (local address 127.0.98.3:37935)
0925 14:50:36.456733 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:36.456735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:36.456793 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:50:36.456867 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:36.456867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:36.456877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:36.456940 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:36.457035 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.457474 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.458390 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:36.458743 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:36.458815 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:36.458957 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:36.458985 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:36.459029 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:36.459084 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:50:36.459100 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:50:36.460088 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:50:37.986898 2134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:37.987027 2134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:37.987785 2134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Sending DeleteTablet for 3 replicas of tablet 41b891b96586407fb79fa9dd2297bc21
I20260925 14:50:37.988026 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b891b96586407fb79fa9dd2297bc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260925 14:50:37.988091 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b891b96586407fb79fa9dd2297bc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260925 14:50:37.988204 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b891b96586407fb79fa9dd2297bc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260925 14:50:37.988371 2723 tablet_replica.cc:330] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: stopping tablet replica
I20260925 14:50:37.988463 2723 raft_consensus.cc:2229] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:37.988536 2723 raft_consensus.cc:2258] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:37.988806 2723 ts_tablet_manager.cc:1905] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:37.989878 2721 tablet_replica.cc:330] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: stopping tablet replica
I20260925 14:50:37.989960 2721 raft_consensus.cc:2229] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:37.990024 2721 raft_consensus.cc:2258] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:37.990248 2721 ts_tablet_manager.cc:1905] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:37.990633 2722 tablet_replica.cc:330] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: stopping tablet replica
I20260925 14:50:37.990707 2722 raft_consensus.cc:2229] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:37.990763 2723 ts_tablet_manager.cc:1918] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:50:37.990844 2723 log.cc:1196] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/41b891b96586407fb79fa9dd2297bc21
I20260925 14:50:37.990855 2722 raft_consensus.cc:2258] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:37.991102 2722 ts_tablet_manager.cc:1905] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:37.991124 2723 ts_tablet_manager.cc:1939] T 41b891b96586407fb79fa9dd2297bc21 P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting consensus metadata
I20260925 14:50:37.991566 2120 catalog_manager.cc:4841] TS a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611): tablet 41b891b96586407fb79fa9dd2297bc21 (table test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]) successfully deleted
I20260925 14:50:37.991786 2721 ts_tablet_manager.cc:1918] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:50:37.991858 2721 log.cc:1196] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/41b891b96586407fb79fa9dd2297bc21
I20260925 14:50:37.992056 2721 ts_tablet_manager.cc:1939] T 41b891b96586407fb79fa9dd2297bc21 P 0cb98b792fdd452d98b764537e472963: Deleting consensus metadata
I20260925 14:50:37.992456 2118 catalog_manager.cc:4841] TS 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329): tablet 41b891b96586407fb79fa9dd2297bc21 (table test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]) successfully deleted
I20260925 14:50:37.992683 2722 ts_tablet_manager.cc:1918] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:50:37.992754 2722 log.cc:1196] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/41b891b96586407fb79fa9dd2297bc21
I20260925 14:50:37.992956 2722 ts_tablet_manager.cc:1939] T 41b891b96586407fb79fa9dd2297bc21 P 87728b96f2b747ce9e82f758c0b15190: Deleting consensus metadata
I20260925 14:50:37.993331 2121 catalog_manager.cc:4841] TS 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935): tablet 41b891b96586407fb79fa9dd2297bc21 (table test-table [id=4fbb42806c9b4e009eb30b5bf4e04853]) successfully deleted
Sep 25 14:50:37 dist-test-slave-gpc1 krb5kdc[2089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:50:38 dist-test-slave-gpc1 krb5kdc[2089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:38.018270 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:38.007296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.0.98.62:34611)
0925 14:50:38.007487 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:38.007489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:38.007499 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:38.007546 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:38.007547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:38.007558 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:38.007626 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:38.008186 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.008759 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.009469 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.009679 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.009749 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:38.015623 (+ 5874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:38.015625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:38.015625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:38.015632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:38.017060 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:38.017402 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:38.017403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:38.017403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:38.017486 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:38.017688 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:38.017689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:38.017690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:38.017806 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:38.017916 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:38.018066 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:50:38.018154 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260925 14:50:38.019860 2134 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:38.020028 2134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:38.021379 2271 tablet_service.cc:1467] Processing CreateTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (DEFAULT_TABLE table=test-table [id=936d1255880b466c9d125c554cf0fcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.021440 2551 tablet_service.cc:1467] Processing CreateTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (DEFAULT_TABLE table=test-table [id=936d1255880b466c9d125c554cf0fcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.021477 2410 tablet_service.cc:1467] Processing CreateTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (DEFAULT_TABLE table=test-table [id=936d1255880b466c9d125c554cf0fcbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.021514 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.021569 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.021584 2410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.022867 2732 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Bootstrap starting.
I20260925 14:50:38.023265 2732 tablet_bootstrap.cc:654] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.023586 2733 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Bootstrap starting.
I20260925 14:50:38.023952 2732 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: No bootstrap required, opened a new log
I20260925 14:50:38.024026 2733 tablet_bootstrap.cc:654] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.024044 2732 ts_tablet_manager.cc:1397] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.024173 2732 raft_consensus.cc:348] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.024272 2732 raft_consensus.cc:374] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.024299 2732 raft_consensus.cc:729] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.024335 2734 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Bootstrap starting.
I20260925 14:50:38.024351 2732 consensus_queue.cc:260] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.024489 2732 ts_tablet_manager.cc:1428] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:38.024641 2733 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: No bootstrap required, opened a new log
I20260925 14:50:38.024715 2733 ts_tablet_manager.cc:1397] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.024755 2734 tablet_bootstrap.cc:654] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.024861 2733 raft_consensus.cc:348] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.024950 2733 raft_consensus.cc:374] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.025007 2733 raft_consensus.cc:729] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb98b792fdd452d98b764537e472963, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.025103 2733 consensus_queue.cc:260] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.025317 2733 ts_tablet_manager.cc:1428] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.025563 2734 tablet_bootstrap.cc:492] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: No bootstrap required, opened a new log
I20260925 14:50:38.025627 2734 ts_tablet_manager.cc:1397] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.025753 2734 raft_consensus.cc:348] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.025856 2734 raft_consensus.cc:374] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.025897 2734 raft_consensus.cc:729] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.025974 2734 consensus_queue.cc:260] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.026081 2734 ts_tablet_manager.cc:1428] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:38.097457 2702 raft_consensus.cc:482] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:38.097579 2702 raft_consensus.cc:504] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.097746 2702 leader_election.cc:288] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:38.097980 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e8b6ae1e3b4b70b15bc9b2f8533cce" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190" is_pre_election: true
I20260925 14:50:38.097973 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e8b6ae1e3b4b70b15bc9b2f8533cce" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963" is_pre_election: true
I20260925 14:50:38.098069 2571 raft_consensus.cc:2454] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:38.098069 2291 raft_consensus.cc:2454] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:38.098227 2365 leader_election.cc:302] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87728b96f2b747ce9e82f758c0b15190, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:38.098315 2702 raft_consensus.cc:2781] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:38.098393 2702 raft_consensus.cc:482] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:38.098421 2702 raft_consensus.cc:3037] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.099392 2702 raft_consensus.cc:504] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.099547 2702 leader_election.cc:288] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Requested vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:38.099659 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e8b6ae1e3b4b70b15bc9b2f8533cce" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963"
I20260925 14:50:38.099730 2291 raft_consensus.cc:3037] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.099714 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4e8b6ae1e3b4b70b15bc9b2f8533cce" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190"
I20260925 14:50:38.099845 2571 raft_consensus.cc:3037] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.100304 2291 raft_consensus.cc:2454] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:38.100478 2362 leader_election.cc:302] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:38.100581 2702 raft_consensus.cc:2781] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:38.100627 2702 raft_consensus.cc:686] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Becoming Leader. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Running, Role: LEADER
I20260925 14:50:38.100615 2571 raft_consensus.cc:2454] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:38.100677 2702 consensus_queue.cc:237] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.101305 2133 catalog_manager.cc:5495] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:38.119632 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:38.117357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.0.98.2:42611)
0925 14:50:38.117529 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:38.117531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:38.117539 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:38.117562 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:38.117567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:38.117577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:38.117648 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:38.117759 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.118224 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.119071 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.119277 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.119329 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:38.119449 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:38.119473 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:38.119508 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:38.119561 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:50:38.119576 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":129,"threads_started":1}
I20260925 14:50:38.121941 2291 raft_consensus.cc:1261] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:38.122008 2571 raft_consensus.cc:1261] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:38.122175 2707 consensus_queue.cc:1035] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:38.122264 2650 consensus_queue.cc:1035] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:38.123570 2703 mvcc.cc:205] Tried to move back new op lower bound from 7333264744946335744 to 7333264744860774400. Current Snapshot: MvccSnapshot[applied={T|T < 7333264744946335744}]
I20260925 14:50:38.132575 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:38.132678 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:38.133301 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Sending DeleteTablet for 3 replicas of tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce
I20260925 14:50:38.133561 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260925 14:50:38.133587 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260925 14:50:38.133659 2721 tablet_replica.cc:330] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: stopping tablet replica
I20260925 14:50:38.133680 2722 tablet_replica.cc:330] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: stopping tablet replica
I20260925 14:50:38.133716 2721 raft_consensus.cc:2229] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:38.133724 2722 raft_consensus.cc:2229] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:38.133757 2721 raft_consensus.cc:2258] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:38.133755 2722 raft_consensus.cc:2258] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:38.133828 2411 tablet_service.cc:1514] Processing DeleteTablet for tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260925 14:50:38.133917 2723 tablet_replica.cc:330] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: stopping tablet replica
I20260925 14:50:38.133983 2723 raft_consensus.cc:2229] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:38.133949 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:50:38.134050 2723 raft_consensus.cc:2258] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 14:50:38.134099 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:38.134147 2722 ts_tablet_manager.cc:1905] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:38.134441 2721 ts_tablet_manager.cc:1905] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:38.134747 2723 ts_tablet_manager.cc:1905] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:38.135739 2722 ts_tablet_manager.cc:1918] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:38.135780 2722 log.cc:1196] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/d4e8b6ae1e3b4b70b15bc9b2f8533cce
I20260925 14:50:38.136003 2722 ts_tablet_manager.cc:1939] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 87728b96f2b747ce9e82f758c0b15190: Deleting consensus metadata
I20260925 14:50:38.136147 2721 ts_tablet_manager.cc:1918] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:38.136199 2721 log.cc:1196] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/d4e8b6ae1e3b4b70b15bc9b2f8533cce
I20260925 14:50:38.136264 2121 catalog_manager.cc:4841] TS 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935): tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (table test-table [id=936d1255880b466c9d125c554cf0fcbe]) successfully deleted
I20260925 14:50:38.136445 2721 ts_tablet_manager.cc:1939] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P 0cb98b792fdd452d98b764537e472963: Deleting consensus metadata
I20260925 14:50:38.136502 2723 ts_tablet_manager.cc:1918] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:50:38.136569 2723 log.cc:1196] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/d4e8b6ae1e3b4b70b15bc9b2f8533cce
I20260925 14:50:38.136799 2723 ts_tablet_manager.cc:1939] T d4e8b6ae1e3b4b70b15bc9b2f8533cce P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting consensus metadata
I20260925 14:50:38.136922 2118 catalog_manager.cc:4841] TS 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329): tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (table test-table [id=936d1255880b466c9d125c554cf0fcbe]) successfully deleted
I20260925 14:50:38.137140 2120 catalog_manager.cc:4841] TS a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611): tablet d4e8b6ae1e3b4b70b15bc9b2f8533cce (table test-table [id=936d1255880b466c9d125c554cf0fcbe]) successfully deleted
I20260925 14:50:38.137447 2271 tablet_service.cc:1467] Processing CreateTablet for tablet 4b21f7434690471695a2aab35e6e3ecc (DEFAULT_TABLE table=test-table [id=e1b36d44e6d440159402e0d9156c887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.137539 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b21f7434690471695a2aab35e6e3ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.138279 2733 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963: Bootstrap starting.
I20260925 14:50:38.138386 2411 tablet_service.cc:1467] Processing CreateTablet for tablet 4b21f7434690471695a2aab35e6e3ecc (DEFAULT_TABLE table=test-table [id=e1b36d44e6d440159402e0d9156c887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.138474 2411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b21f7434690471695a2aab35e6e3ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.138767 2733 tablet_bootstrap.cc:654] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.139262 2734 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Bootstrap starting.
I20260925 14:50:38.139429 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 4b21f7434690471695a2aab35e6e3ecc (DEFAULT_TABLE table=test-table [id=e1b36d44e6d440159402e0d9156c887e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:38.139520 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b21f7434690471695a2aab35e6e3ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.139595 2734 tablet_bootstrap.cc:654] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.139725 2733 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963: No bootstrap required, opened a new log
I20260925 14:50:38.139771 2733 ts_tablet_manager.cc:1397] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:38.139925 2733 raft_consensus.cc:348] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.140033 2733 raft_consensus.cc:374] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.140069 2733 raft_consensus.cc:729] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb98b792fdd452d98b764537e472963, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.140121 2733 consensus_queue.cc:260] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.140158 2734 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: No bootstrap required, opened a new log
I20260925 14:50:38.140199 2734 ts_tablet_manager.cc:1397] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.140231 2733 ts_tablet_manager.cc:1428] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:38.140311 2734 raft_consensus.cc:348] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.140385 2734 raft_consensus.cc:374] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.140408 2734 raft_consensus.cc:729] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.140460 2734 consensus_queue.cc:260] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.140568 2734 ts_tablet_manager.cc:1428] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:38.140672 2732 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Bootstrap starting.
I20260925 14:50:38.141014 2732 tablet_bootstrap.cc:654] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.141666 2732 tablet_bootstrap.cc:492] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: No bootstrap required, opened a new log
I20260925 14:50:38.141709 2732 ts_tablet_manager.cc:1397] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:38.141824 2732 raft_consensus.cc:348] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.141894 2732 raft_consensus.cc:374] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.141911 2732 raft_consensus.cc:729] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87728b96f2b747ce9e82f758c0b15190, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.141980 2732 consensus_queue.cc:260] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.142089 2732 ts_tablet_manager.cc:1428] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:38.163827 2702 raft_consensus.cc:482] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:38.163925 2702 raft_consensus.cc:504] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.164080 2702 leader_election.cc:288] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:38.164258 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b21f7434690471695a2aab35e6e3ecc" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963" is_pre_election: true
I20260925 14:50:38.164287 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b21f7434690471695a2aab35e6e3ecc" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190" is_pre_election: true
I20260925 14:50:38.164376 2571 raft_consensus.cc:2454] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:38.164422 2291 raft_consensus.cc:2454] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 0.
I20260925 14:50:38.164563 2365 leader_election.cc:302] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87728b96f2b747ce9e82f758c0b15190, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:38.164670 2702 raft_consensus.cc:2781] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:38.164722 2702 raft_consensus.cc:482] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:38.164754 2702 raft_consensus.cc:3037] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.165437 2702 raft_consensus.cc:504] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.165588 2702 leader_election.cc:288] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Requested vote from peers 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329), 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935)
I20260925 14:50:38.165756 2291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b21f7434690471695a2aab35e6e3ecc" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb98b792fdd452d98b764537e472963"
I20260925 14:50:38.165755 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b21f7434690471695a2aab35e6e3ecc" candidate_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87728b96f2b747ce9e82f758c0b15190"
I20260925 14:50:38.165851 2571 raft_consensus.cc:3037] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.165855 2291 raft_consensus.cc:3037] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.166512 2291 raft_consensus.cc:2454] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:38.166642 2571 raft_consensus.cc:2454] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a06b64eb2c024ce78245d4d4d0ec68ac in term 1.
I20260925 14:50:38.166759 2362 leader_election.cc:302] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb98b792fdd452d98b764537e472963, a06b64eb2c024ce78245d4d4d0ec68ac; no voters:
I20260925 14:50:38.166848 2702 raft_consensus.cc:2781] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:38.166920 2702 raft_consensus.cc:686] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Becoming Leader. State: Replica: a06b64eb2c024ce78245d4d4d0ec68ac, State: Running, Role: LEADER
I20260925 14:50:38.166981 2702 consensus_queue.cc:237] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } }
I20260925 14:50:38.167572 2133 catalog_manager.cc:5495] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac reported cstate change: term changed from 0 to 1, leader changed from <none> to a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a06b64eb2c024ce78245d4d4d0ec68ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:38.173162 2571 raft_consensus.cc:1261] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:38.173187 2291 raft_consensus.cc:1261] T 4b21f7434690471695a2aab35e6e3ecc P 0cb98b792fdd452d98b764537e472963 [term 1 FOLLOWER]: Refusing update from remote peer a06b64eb2c024ce78245d4d4d0ec68ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:38.173441 2749 consensus_queue.cc:1035] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "87728b96f2b747ce9e82f758c0b15190" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:38.173556 2707 consensus_queue.cc:1035] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb98b792fdd452d98b764537e472963" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:50:38.178192 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:50:39.697461 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:39.697573 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:39.698401 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05ad4a47eed84cde85dac56ad961578b: Sending DeleteTablet for 3 replicas of tablet 4b21f7434690471695a2aab35e6e3ecc
I20260925 14:50:39.698650 2410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b21f7434690471695a2aab35e6e3ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260925 14:50:39.698663 2271 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b21f7434690471695a2aab35e6e3ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260925 14:50:39.698793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2206
I20260925 14:50:39.698832 2774 tablet_replica.cc:330] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: stopping tablet replica
I20260925 14:50:39.698871 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b21f7434690471695a2aab35e6e3ecc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54856
I20260925 14:50:39.700120 2776 tablet_replica.cc:330] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: stopping tablet replica
I20260925 14:50:39.700184 2776 raft_consensus.cc:2229] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:39.700212 2776 raft_consensus.cc:2258] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:39.700489 2776 ts_tablet_manager.cc:1905] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:39.698882 2774 raft_consensus.cc:2229] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:39.700924 2774 raft_consensus.cc:2258] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:39.701160 2774 ts_tablet_manager.cc:1905] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:39.702492 2774 ts_tablet_manager.cc:1918] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:50:39.702508 2776 ts_tablet_manager.cc:1918] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:50:39.702564 2776 log.cc:1196] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/4b21f7434690471695a2aab35e6e3ecc
I20260925 14:50:39.702562 2774 log.cc:1196] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/4b21f7434690471695a2aab35e6e3ecc
I20260925 14:50:39.702806 2774 ts_tablet_manager.cc:1939] T 4b21f7434690471695a2aab35e6e3ecc P a06b64eb2c024ce78245d4d4d0ec68ac: Deleting consensus metadata
I20260925 14:50:39.702849 2776 ts_tablet_manager.cc:1939] T 4b21f7434690471695a2aab35e6e3ecc P 87728b96f2b747ce9e82f758c0b15190: Deleting consensus metadata
I20260925 14:50:39.703136 2120 catalog_manager.cc:4841] TS a06b64eb2c024ce78245d4d4d0ec68ac (127.0.98.2:42611): tablet 4b21f7434690471695a2aab35e6e3ecc (table test-table [id=e1b36d44e6d440159402e0d9156c887e]) successfully deleted
I20260925 14:50:39.703274 2121 catalog_manager.cc:4841] TS 87728b96f2b747ce9e82f758c0b15190 (127.0.98.3:37935): tablet 4b21f7434690471695a2aab35e6e3ecc (table test-table [id=e1b36d44e6d440159402e0d9156c887e]) successfully deleted
W20260925 14:50:39.704344 2118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41329 (error 108)
I20260925 14:50:39.704851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2346
I20260925 14:50:39.704891 2778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.704683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41329 (local address 127.0.0.1:52772)
0925 14:50:39.704805 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.704852 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
W20260925 14:50:39.704993 2118 catalog_manager.cc:4584] TS 0cb98b792fdd452d98b764537e472963 (127.0.98.1:41329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b21f7434690471695a2aab35e6e3ecc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41329: connect: Connection refused (error 111)
I20260925 14:50:39.711349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2486
I20260925 14:50:39.717585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2102
2026-09-25T14:50:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5829 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:39 dist-test-slave-gpc1 krb5kdc[2781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:39 dist-test-slave-gpc1 krb5kdc[2781](info): set up 2 sockets
Sep 25 14:50:39 dist-test-slave-gpc1 krb5kdc[2781](info): commencing operation
krb5kdc: starting...
W20260925 14:50:40.754175 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:40 dist-test-slave-gpc1 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:40.870690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:40.949687 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:40.949923 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:40.949961 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:40.951182 2794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:40.951246 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:40.951270 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:40.951299 2794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:40.951331 2794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:40.952829 2794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:40.953116 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:40.953330 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:40.958338 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:40.958510 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:40.958480 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:40.958527 2794 server_base.cc:1034] running on GCE node
I20260925 14:50:40.958877 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:40.959174 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:40.960353 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790347840960333 us; error 38 us; skew 500 ppm
Sep 25 14:50:40 dist-test-slave-gpc1 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:40.962901 2794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:40.963366 2794 webserver.cc:460] Webserver started at http://127.0.98.62:35729/ using document root <none> and password file <none>
I20260925 14:50:40.963599 2794 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:40.963663 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:40.963805 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:40.964793 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "be85ffc3de0a41b5b33719a0a42c66da"
format_stamp: "Formatted at 2026-09-25 14:50:40 on dist-test-slave-gpc1"
server_key: "052917e215a5bc07574fa2a39fcfd3d1"
server_key_iv: "4bac1e979dfed5d85767e31477daf7d4"
server_key_version: "encryptionkey@0"
I20260925 14:50:40.965133 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "be85ffc3de0a41b5b33719a0a42c66da"
format_stamp: "Formatted at 2026-09-25 14:50:40 on dist-test-slave-gpc1"
server_key: "052917e215a5bc07574fa2a39fcfd3d1"
server_key_iv: "4bac1e979dfed5d85767e31477daf7d4"
server_key_version: "encryptionkey@0"
I20260925 14:50:40.966528 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:40.967224 2809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:40.967427 2794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:40.967553 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "be85ffc3de0a41b5b33719a0a42c66da"
format_stamp: "Formatted at 2026-09-25 14:50:40 on dist-test-slave-gpc1"
server_key: "052917e215a5bc07574fa2a39fcfd3d1"
server_key_iv: "4bac1e979dfed5d85767e31477daf7d4"
server_key_version: "encryptionkey@0"
I20260925 14:50:40.967653 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:40.976992 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:40.978838 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:40.978997 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:40.982928 2794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35867
I20260925 14:50:40.982961 2861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35867 every 8 connection(s)
I20260925 14:50:40.983470 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:40.984093 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:40.985066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2794
I20260925 14:50:40.985157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:40.985334 392 external_mini_cluster.cc:1428] Setting key 2f033dc83f8f962d7d658889b5e5f9fb
I20260925 14:50:40.986860 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da: Bootstrap starting.
I20260925 14:50:40.987490 2862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:40.987804 2862 log.cc:824] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:40.988509 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da: No bootstrap required, opened a new log
I20260925 14:50:40.990747 2862 raft_consensus.cc:348] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260925 14:50:40.990914 2862 raft_consensus.cc:374] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:40.990963 2862 raft_consensus.cc:729] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be85ffc3de0a41b5b33719a0a42c66da, State: Initialized, Role: FOLLOWER
I20260925 14:50:40.991091 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260925 14:50:40.991173 2862 raft_consensus.cc:388] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:40.991216 2862 raft_consensus.cc:482] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:40.991266 2862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:40.991791 2862 raft_consensus.cc:504] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
I20260925 14:50:40.991915 2862 leader_election.cc:302] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be85ffc3de0a41b5b33719a0a42c66da; no voters:
I20260925 14:50:40.992103 2862 leader_election.cc:288] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:40.992136 2867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:40.992304 2867 raft_consensus.cc:686] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [term 1 LEADER]: Becoming Leader. State: Replica: be85ffc3de0a41b5b33719a0a42c66da, State: Running, Role: LEADER
I20260925 14:50:40.992362 2862 sys_catalog.cc:564] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:40.992440 2867 consensus_queue.cc:237] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } }
Sep 25 14:50:40 dist-test-slave-gpc1 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:40.993356 2869 sys_catalog.cc:455] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [sys.catalog]: SysCatalogTable state changed. Reason: New leader be85ffc3de0a41b5b33719a0a42c66da. Latest consensus state: current_term: 1 leader_uuid: "be85ffc3de0a41b5b33719a0a42c66da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } } }
I20260925 14:50:40.993762 2869 sys_catalog.cc:458] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:40.994199 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be85ffc3de0a41b5b33719a0a42c66da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be85ffc3de0a41b5b33719a0a42c66da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35867 } } }
I20260925 14:50:40.994282 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:40.994004 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:40.994870 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:40.996778 2881 catalog_manager.cc:1347] Generated new cluster ID: 8dad01ed139c4f81b1b92dea98bfdbc3
I20260925 14:50:40.996866 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:40.997046 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:40.985842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.0.98.62:35867)
0925 14:50:40.986030 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:40.986039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:40.986072 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:50:40.986648 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:40.986657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:40.986715 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:40.986867 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:40.987487 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:40.988823 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:40.989568 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:40.990272 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:40.990435 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:40.993088 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:40.993092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:40.993092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:40.993104 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:40.995199 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:40.995587 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:40.995590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:40.995590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:40.995688 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:40.995935 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:40.995936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:40.995936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:40.996245 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:40.996366 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:40.996596 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 14:50:40.996703 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260925 14:50:41.004300 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:41.005009 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:41.007668 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be85ffc3de0a41b5b33719a0a42c66da: Generated new TSK 0
I20260925 14:50:41.007822 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:41.035398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:41.116555 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:41.116739 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:41.116760 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:41.117897 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:41.117935 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:41.117961 2888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:41.119550 2888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:41.119788 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:41.120018 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:41.125069 2893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:41.125037 2896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:41.125051 2894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:41.125155 2888 server_base.cc:1034] running on GCE node
I20260925 14:50:41.125443 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:41.125752 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:41.126899 2888 hybrid_clock.cc:648] HybridClock initialized: now 1790347841126891 us; error 39 us; skew 500 ppm
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:41.129511 2888 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:41.129931 2888 webserver.cc:460] Webserver started at http://127.0.98.1:36539/ using document root <none> and password file <none>
I20260925 14:50:41.130185 2888 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:41.130259 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:41.130434 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:41.131453 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "5eb4c1d73aff4a4d8e9c86cdfc0a0dd5"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "39ed1f792d4c1d03937557169b7ee1ee"
server_key_iv: "c04d093eaae1cafe6974adef1fbe0b87"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.131806 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5eb4c1d73aff4a4d8e9c86cdfc0a0dd5"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "39ed1f792d4c1d03937557169b7ee1ee"
server_key_iv: "c04d093eaae1cafe6974adef1fbe0b87"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.133140 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:50:41.134011 2903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.134186 2888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:41.134388 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "5eb4c1d73aff4a4d8e9c86cdfc0a0dd5"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "39ed1f792d4c1d03937557169b7ee1ee"
server_key_iv: "c04d093eaae1cafe6974adef1fbe0b87"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.134491 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:41.156299 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:41.158440 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:41.158615 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:41.158878 2888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:41.159214 2888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:41.159273 2888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.159322 2888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:41.159350 2888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.165607 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42475
I20260925 14:50:41.165686 3016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42475 every 8 connection(s)
I20260925 14:50:41.166029 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:41.170130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 2888
I20260925 14:50:41.170221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:41.170454 392 external_mini_cluster.cc:1428] Setting key 13c7355307663729b95f7d3cb154cbc4
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:41.176481 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.166798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48725 (local address 127.0.98.62:35867)
0925 14:50:41.166871 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:41.166873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:41.167073 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:50:41.167993 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:41.167994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:41.168004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:41.168070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:41.168681 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.169246 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.170461 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.170812 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.170881 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.173334 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:41.173335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:41.173336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:41.173342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:41.175047 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.175485 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.175487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.175487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.175551 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.175798 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.175800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.175800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.175922 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:41.175992 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:41.176275 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 14:50:41.176369 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:50:41.176733 3019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.166777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35867 (local address 127.0.98.1:48725)
0925 14:50:41.166944 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:41.166985 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:41.167897 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:41.168127 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:41.168131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:41.168378 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:41.168568 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.168571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.169325 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.169326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.170329 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.170332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.170894 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.170895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.171059 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.171686 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:41.171694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:41.173236 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:41.175144 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.175151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.175153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.175391 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.175616 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.175617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.175618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.175732 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.175994 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:41.176001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:41.176147 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:50:41.176254 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 14:50:41.176423 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260925 14:50:41.177211 3017 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35867
I20260925 14:50:41.177306 3017 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:41.177513 3017 heartbeater.cc:511] Master 127.0.98.62:35867 requested a full tablet report, sending...
I20260925 14:50:41.178028 2826 ts_manager.cc:194] Registered new tserver with Master: 5eb4c1d73aff4a4d8e9c86cdfc0a0dd5 (127.0.98.1:42475)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:50:41.178817 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:41.196551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:41.276767 3022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:41.277062 3022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:41.277102 3022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:41.278273 3022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:41.278339 3022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:41.278414 3022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:41.279937 3022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:41.280192 3022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:41.280421 3022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:41.285372 3027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:41.285457 3030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:41.285604 3022 server_base.cc:1034] running on GCE node
W20260925 14:50:41.285493 3028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:41.285858 3022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:41.286068 3022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:41.287205 3022 hybrid_clock.cc:648] HybridClock initialized: now 1790347841287149 us; error 72 us; skew 500 ppm
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:41.289553 3022 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:41.289991 3022 webserver.cc:460] Webserver started at http://127.0.98.2:40551/ using document root <none> and password file <none>
I20260925 14:50:41.290167 3022 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:41.290207 3022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:41.290293 3022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:41.291304 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "af470db23a2747758b2439661f57f459"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "7e5a6ead7d2250bffaf7f289c768d536"
server_key_iv: "3727bfb966d7d20bcfc1f0ef7213da15"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.291602 3022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af470db23a2747758b2439661f57f459"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "7e5a6ead7d2250bffaf7f289c768d536"
server_key_iv: "3727bfb966d7d20bcfc1f0ef7213da15"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.292999 3022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:41.293671 3037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.293843 3022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:41.293928 3022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "af470db23a2747758b2439661f57f459"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "7e5a6ead7d2250bffaf7f289c768d536"
server_key_iv: "3727bfb966d7d20bcfc1f0ef7213da15"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.294023 3022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:41.303618 3022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:41.305415 3022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:41.305567 3022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:41.305766 3022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:41.306066 3022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:41.306119 3022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.306157 3022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:41.306186 3022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.312006 3022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42541
I20260925 14:50:41.312052 3150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42541 every 8 connection(s)
I20260925 14:50:41.312371 3022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:41.321117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3022
I20260925 14:50:41.321226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:41.321487 392 external_mini_cluster.cc:1428] Setting key 5470448757087a95d0ddd8a3ed42ff1c
I20260925 14:50:41.321647 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.313114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58849 (local address 127.0.98.62:35867)
0925 14:50:41.313178 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:41.313180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:41.313401 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:50:41.314391 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:41.314392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:41.314404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:41.314474 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:41.315095 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.315607 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.316601 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.316801 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.316877 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.318942 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:41.318943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:41.318944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:41.318950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:41.320149 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.320562 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.320564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.320564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.320623 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.320892 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.320893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.320894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.321009 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:41.321121 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:41.321477 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 14:50:41.321558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:50:41.321843 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.313105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35867 (local address 127.0.98.2:58849)
0925 14:50:41.313249 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:41.313281 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:41.314263 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:41.314562 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:41.314565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:41.314812 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:41.314998 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.315001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.315700 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.315702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.316508 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.316510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.316884 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.316885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.317007 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.317457 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:41.317464 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:41.318829 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:41.320241 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.320247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.320249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.320476 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.320693 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.320695 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.320695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.320798 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.321115 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:41.321121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:41.321309 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 14:50:41.321438 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:50:41.321608 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260925 14:50:41.322449 3151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35867
I20260925 14:50:41.322551 3151 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:41.322736 3151 heartbeater.cc:511] Master 127.0.98.62:35867 requested a full tablet report, sending...
I20260925 14:50:41.323129 2826 ts_manager.cc:194] Registered new tserver with Master: af470db23a2747758b2439661f57f459 (127.0.98.2:42541)
I20260925 14:50:41.323624 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58849
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:41.345280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:41.424938 3156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:41.425160 3156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:41.425199 3156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:41.426450 3156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:41.426510 3156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:41.426551 3156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:41.428030 3156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:41.428300 3156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:41.428539 3156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:41.433398 3162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:41.433473 3164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:41.433527 3156 server_base.cc:1034] running on GCE node
W20260925 14:50:41.433492 3161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:41.433812 3156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:41.434078 3156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:41.435258 3156 hybrid_clock.cc:648] HybridClock initialized: now 1790347841435242 us; error 39 us; skew 500 ppm
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:41.437736 3156 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:41.438194 3156 webserver.cc:460] Webserver started at http://127.0.98.3:44309/ using document root <none> and password file <none>
I20260925 14:50:41.438498 3156 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:41.438566 3156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:41.438697 3156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:41.439683 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "76c4ba64ac74486a811ba1ac4355306e"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "4a99235fe2a8f7e329b15c6b7ec3238e"
server_key_iv: "c8f75f3ea0a24f8c0fa6b9333f2c573f"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.440064 3156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76c4ba64ac74486a811ba1ac4355306e"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "4a99235fe2a8f7e329b15c6b7ec3238e"
server_key_iv: "c8f75f3ea0a24f8c0fa6b9333f2c573f"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.441326 3156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:41.441988 3171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.442157 3156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:41.442251 3156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "76c4ba64ac74486a811ba1ac4355306e"
format_stamp: "Formatted at 2026-09-25 14:50:41 on dist-test-slave-gpc1"
server_key: "4a99235fe2a8f7e329b15c6b7ec3238e"
server_key_iv: "c8f75f3ea0a24f8c0fa6b9333f2c573f"
server_key_version: "encryptionkey@0"
I20260925 14:50:41.442337 3156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:41.450866 3156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:41.452678 3156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:41.452836 3156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:41.453047 3156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:41.453330 3156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:41.453385 3156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.453418 3156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:41.453445 3156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:41.459033 3156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44833
I20260925 14:50:41.459084 3284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44833 every 8 connection(s)
I20260925 14:50:41.459362 3156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[2781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:41.468034 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.460006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38089 (local address 127.0.98.62:35867)
0925 14:50:41.460087 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:41.460090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:41.460223 (+ 133us) server_negotiation.cc:407] Connection header received
0925 14:50:41.461156 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:41.461157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:41.461169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:41.461231 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:41.461810 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.462316 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.463344 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.463562 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.463635 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.465663 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:41.465664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:41.465664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:41.465670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:41.466910 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.467265 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.467266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.467266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.467319 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:41.467546 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:41.467547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:41.467547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:41.467646 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:41.467695 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:41.467912 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:50:41.467970 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:50:41.468246 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:41.459957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35867 (local address 127.0.98.3:38089)
0925 14:50:41.460106 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:41.460137 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:41.461045 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:41.461294 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:41.461300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:41.461543 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:41.461716 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.461720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.462454 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.462455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.463251 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:41.463253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:41.463651 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:41.463652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:41.463774 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:41.464283 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:41.464291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:41.465574 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:41.466982 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.466987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.466988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.467189 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.467397 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:41.467398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:41.467398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:41.467491 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:41.467699 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:41.467703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:41.467808 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 14:50:41.467897 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:50:41.468008 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260925 14:50:41.468717 3285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35867
I20260925 14:50:41.468794 3285 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:41.468988 3285 heartbeater.cc:511] Master 127.0.98.62:35867 requested a full tablet report, sending...
I20260925 14:50:41.469421 2826 ts_manager.cc:194] Registered new tserver with Master: 76c4ba64ac74486a811ba1ac4355306e (127.0.98.3:44833)
I20260925 14:50:41.469532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3156
I20260925 14:50:41.469627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:41.469820 392 external_mini_cluster.cc:1428] Setting key 60b30975c882ddc9039b764154e909a4
I20260925 14:50:41.469946 2826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38089
I20260925 14:50:41.470748 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:50:41.481660 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:41.480732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54834 (local address 127.0.98.62:35867)
0925 14:50:41.480808 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:41.480811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:41.480929 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:50:41.480974 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:41.480975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:41.480985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:41.481035 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:41.481587 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54834: BlockingRecv error: recv got EOF from 127.0.0.1:54834 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:50:41.483039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2888
I20260925 14:50:41.487215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3022
I20260925 14:50:41.491299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3156
I20260925 14:50:41.495678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 2794
2026-09-25T14:50:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1776 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[3297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[3297](info): set up 2 sockets
Sep 25 14:50:41 dist-test-slave-gpc1 krb5kdc[3297](info): commencing operation
krb5kdc: starting...
W20260925 14:50:42.534297 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:42.650112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:42.728531 3310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:42.728754 3310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:42.728792 3310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:42.729943 3310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:42.730005 3310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:42.730028 3310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:42.730057 3310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:42.730080 3310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:42.731549 3310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:42.731809 3310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:42.732033 3310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:42.736713 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:42.736730 3316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:42.736748 3318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:42.736984 3310 server_base.cc:1034] running on GCE node
I20260925 14:50:42.737231 3310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:42.737555 3310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:42.738726 3310 hybrid_clock.cc:648] HybridClock initialized: now 1790347842738701 us; error 36 us; skew 500 ppm
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:42.741134 3310 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:42.741570 3310 webserver.cc:460] Webserver started at http://127.0.98.62:35265/ using document root <none> and password file <none>
I20260925 14:50:42.741801 3310 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:42.741869 3310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:42.742014 3310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:42.743116 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "661d6f2346fe4e7d96db81ac46166492"
format_stamp: "Formatted at 2026-09-25 14:50:42 on dist-test-slave-gpc1"
server_key: "f9c830605fcb3cf6542e0fd258772e45"
server_key_iv: "d2f2cf7a3f404412c13f342c5b0af0a4"
server_key_version: "encryptionkey@0"
I20260925 14:50:42.743461 3310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "661d6f2346fe4e7d96db81ac46166492"
format_stamp: "Formatted at 2026-09-25 14:50:42 on dist-test-slave-gpc1"
server_key: "f9c830605fcb3cf6542e0fd258772e45"
server_key_iv: "d2f2cf7a3f404412c13f342c5b0af0a4"
server_key_version: "encryptionkey@0"
I20260925 14:50:42.744844 3310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:50:42.745610 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:42.745781 3310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:42.745888 3310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "661d6f2346fe4e7d96db81ac46166492"
format_stamp: "Formatted at 2026-09-25 14:50:42 on dist-test-slave-gpc1"
server_key: "f9c830605fcb3cf6542e0fd258772e45"
server_key_iv: "d2f2cf7a3f404412c13f342c5b0af0a4"
server_key_version: "encryptionkey@0"
I20260925 14:50:42.745967 3310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:42.762810 3310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:42.765175 3310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:42.765354 3310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:42.769408 3310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40655
I20260925 14:50:42.769469 3377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40655 every 8 connection(s)
I20260925 14:50:42.769834 3310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:42.770761 3378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:42.772976 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492: Bootstrap starting.
I20260925 14:50:42.773500 3378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:42.773733 3378 log.cc:824] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:42.774406 3378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492: No bootstrap required, opened a new log
I20260925 14:50:42.774564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3310
I20260925 14:50:42.774650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:42.774811 392 external_mini_cluster.cc:1428] Setting key d3e21a4a75e116dc7e0425f8725d046f
I20260925 14:50:42.776759 3378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260925 14:50:42.776896 3378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:42.776919 3378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661d6f2346fe4e7d96db81ac46166492, State: Initialized, Role: FOLLOWER
I20260925 14:50:42.777112 3378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260925 14:50:42.777199 3378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:42.777236 3378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:42.777288 3378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:42.777937 3378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260925 14:50:42.778020 3378 leader_election.cc:302] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 661d6f2346fe4e7d96db81ac46166492; no voters:
I20260925 14:50:42.778211 3378 leader_election.cc:288] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:42.778242 3383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:42.778371 3383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [term 1 LEADER]: Becoming Leader. State: Replica: 661d6f2346fe4e7d96db81ac46166492, State: Running, Role: LEADER
I20260925 14:50:42.778452 3383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260925 14:50:42.778484 3378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:42.778761 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "661d6f2346fe4e7d96db81ac46166492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260925 14:50:42.778931 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:42.779119 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 661d6f2346fe4e7d96db81ac46166492. Latest consensus state: current_term: 1 leader_uuid: "661d6f2346fe4e7d96db81ac46166492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "661d6f2346fe4e7d96db81ac46166492" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260925 14:50:42.779232 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:42.779536 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:42.780062 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:42.781760 3391 catalog_manager.cc:1347] Generated new cluster ID: eaf9ca82b37344a78b8024513928d04a
I20260925 14:50:42.781819 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:42.787438 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:42.775293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.98.62:40655)
0925 14:50:42.775469 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:42.775478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:42.775511 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:50:42.776748 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:42.776755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:42.776817 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:42.776988 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:42.777738 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:42.779076 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:42.780223 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:42.780816 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:42.780926 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:42.783674 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:42.783678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:42.783678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:42.783690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:42.785682 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:42.786073 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:42.786076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:42.786076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:42.786200 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:42.786419 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:42.786420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:42.786420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:42.786737 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:42.786911 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:42.787008 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:50:42.787162 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260925 14:50:42.788396 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:50:42.791446 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:42.791926 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:42.798158 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 661d6f2346fe4e7d96db81ac46166492: Generated new TSK 0
I20260925 14:50:42.798331 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:50:42.804926 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:42.800486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.0.98.62:40655)
0925 14:50:42.800535 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:42.800537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:42.800616 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:50:42.800702 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:42.800703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:42.800714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:42.800772 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:42.801171 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:42.801657 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:42.802979 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:42.803352 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:42.803425 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:42.804294 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:42.804295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:42.804295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:42.804568 (+ 273us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:42.804587 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:42.804595 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:42.804669 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:42.804772 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:50:42.804804 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:50:42.807708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3310
2026-09-25T14:50:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1310 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3411](info): set up 2 sockets
Sep 25 14:50:42 dist-test-slave-gpc1 krb5kdc[3411](info): commencing operation
krb5kdc: starting...
W20260925 14:50:43.858333 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:43 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:43.975116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:44.054258 3424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:44.054488 3424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:44.054510 3424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:44.055572 3424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:44.055603 3424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:44.055617 3424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:44.055630 3424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:44.055642 3424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:44.056979 3424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:44.057163 3424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:44.057320 3424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:44.061993 3429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.061997 3430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.062014 3432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:44.062134 3424 server_base.cc:1034] running on GCE node
I20260925 14:50:44.062491 3424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:44.062781 3424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:44.063933 3424 hybrid_clock.cc:648] HybridClock initialized: now 1790347844063915 us; error 30 us; skew 500 ppm
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:44.066464 3424 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:44.066890 3424 webserver.cc:460] Webserver started at http://127.0.98.62:41243/ using document root <none> and password file <none>
I20260925 14:50:44.067111 3424 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:44.067155 3424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:44.067296 3424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:44.068310 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "69ddc2525c4a42769d3d160089308b8d"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "ce8095675ba2febb63ccde151de6c332"
server_key_iv: "57810a04009a4b29b8228122c1dfed73"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.068671 3424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "69ddc2525c4a42769d3d160089308b8d"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "ce8095675ba2febb63ccde151de6c332"
server_key_iv: "57810a04009a4b29b8228122c1dfed73"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.069984 3424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:44.070776 3439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.070947 3424 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.071043 3424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "69ddc2525c4a42769d3d160089308b8d"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "ce8095675ba2febb63ccde151de6c332"
server_key_iv: "57810a04009a4b29b8228122c1dfed73"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.071123 3424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:44.086885 3424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:44.089172 3424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:44.089332 3424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:44.093431 3424 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35749
I20260925 14:50:44.093480 3491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35749 every 8 connection(s)
I20260925 14:50:44.093850 3424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:44.094846 3492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:44.096997 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: Bootstrap starting.
I20260925 14:50:44.097522 3492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:44.097761 3492 log.cc:824] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:44.098457 3492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: No bootstrap required, opened a new log
I20260925 14:50:44.099647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3424
I20260925 14:50:44.099752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:44.099951 392 external_mini_cluster.cc:1428] Setting key e4aabf4d7188d49149e6f43f37cce918
I20260925 14:50:44.100369 3492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } }
I20260925 14:50:44.100556 3492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:44.100611 3492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ddc2525c4a42769d3d160089308b8d, State: Initialized, Role: FOLLOWER
I20260925 14:50:44.100733 3492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } }
I20260925 14:50:44.100816 3492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:44.100848 3492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:44.100893 3492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:44.101888 3492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } }
I20260925 14:50:44.101989 3492 leader_election.cc:302] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ddc2525c4a42769d3d160089308b8d; no voters:
I20260925 14:50:44.102139 3492 leader_election.cc:288] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:44.102252 3497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:44.102437 3497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [term 1 LEADER]: Becoming Leader. State: Replica: 69ddc2525c4a42769d3d160089308b8d, State: Running, Role: LEADER
I20260925 14:50:44.102435 3492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:44.102561 3497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } }
I20260925 14:50:44.102933 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ddc2525c4a42769d3d160089308b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } } }
I20260925 14:50:44.103039 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:44.103773 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ddc2525c4a42769d3d160089308b8d. Latest consensus state: current_term: 1 leader_uuid: "69ddc2525c4a42769d3d160089308b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ddc2525c4a42769d3d160089308b8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35749 } } }
I20260925 14:50:44.103875 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d [sys.catalog]: This master's current role is: LEADER
W20260925 14:50:44.104508 3513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:44.104565 3513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:44.104619 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:44.104766 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:44.106230 3503 catalog_manager.cc:1347] Generated new cluster ID: 9d5d9588f6da40fc9ce4d2688df3732d
I20260925 14:50:44.106279 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:44.111379 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.100468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.0.98.62:35749)
0925 14:50:44.100657 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:44.100666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:44.100694 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:50:44.100803 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:44.100811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:44.100865 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:44.101039 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:44.101592 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.102959 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.104039 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.104648 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.104754 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.107722 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:44.107726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:44.107726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:44.107737 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:44.109646 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.110060 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.110063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.110063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.110169 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.110421 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.110421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.110422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.110696 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:44.110833 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:44.110971 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:50:44.111184 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260925 14:50:44.113559 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:44.114252 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:44.126088 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ddc2525c4a42769d3d160089308b8d: Generated new TSK 0
I20260925 14:50:44.126284 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:44.157315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:44.236296 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:44.236500 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:44.236522 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:44.237682 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:44.237721 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:44.237774 3518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:44.239219 3518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:44.239465 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:44.239717 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:44.244597 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:44.244740 3518 server_base.cc:1034] running on GCE node
W20260925 14:50:44.244729 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.244694 3526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:44.245100 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:44.245306 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:44.246418 3518 hybrid_clock.cc:648] HybridClock initialized: now 1790347844246400 us; error 33 us; skew 500 ppm
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:44.248905 3518 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:44.249337 3518 webserver.cc:460] Webserver started at http://127.0.98.1:36543/ using document root <none> and password file <none>
I20260925 14:50:44.249580 3518 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:44.249650 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:44.249801 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:44.250916 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "917d396fa2334e068c36f5e3797899d2"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "cec946d6c613897da65bed3e927a5152"
server_key_iv: "24f067747311b00f0ebb7ff7fafefca2"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.251300 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "917d396fa2334e068c36f5e3797899d2"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "cec946d6c613897da65bed3e927a5152"
server_key_iv: "24f067747311b00f0ebb7ff7fafefca2"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.252645 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:44.253381 3533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.253551 3518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:44.253646 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "917d396fa2334e068c36f5e3797899d2"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "cec946d6c613897da65bed3e927a5152"
server_key_iv: "24f067747311b00f0ebb7ff7fafefca2"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.253727 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:44.263453 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:44.265249 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:44.265408 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:44.265627 3518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:44.265929 3518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:44.265981 3518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.266014 3518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:44.266040 3518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.271759 3518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35345
I20260925 14:50:44.271898 3646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35345 every 8 connection(s)
I20260925 14:50:44.272187 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:44.281160 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.272895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47227 (local address 127.0.98.62:35749)
0925 14:50:44.272975 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:44.272977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:44.273188 (+ 211us) server_negotiation.cc:407] Connection header received
0925 14:50:44.273764 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:44.273765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:44.273781 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:44.273837 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:44.274923 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.275451 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.276440 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.276653 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.276727 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.278830 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:44.278831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:44.278832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:44.278838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:44.280053 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.280408 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.280409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.280409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.280456 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.280665 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.280665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.280666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.280757 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:44.280806 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:44.281032 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:50:44.281096 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:50:44.281325 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.272883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35749 (local address 127.0.98.1:47227)
0925 14:50:44.273050 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:44.273101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:44.273638 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:44.273907 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:44.273910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:44.274621 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:44.274816 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.274820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.275542 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.275543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.276338 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.276340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.276744 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.276745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.276860 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.277387 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:44.277395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:44.278717 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:44.280122 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.280127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.280129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.280344 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.280510 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.280510 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.280511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.280610 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.280814 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:44.280818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:44.280926 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:50:44.281006 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:50:44.281115 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260925 14:50:44.281858 3647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35749
I20260925 14:50:44.281937 3647 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:44.282114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3518
I20260925 14:50:44.282161 3647 heartbeater.cc:511] Master 127.0.98.62:35749 requested a full tablet report, sending...
I20260925 14:50:44.282212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:44.282498 392 external_mini_cluster.cc:1428] Setting key e4e36cfcec39a3578c71c714b8507b78
I20260925 14:50:44.282788 3456 ts_manager.cc:194] Registered new tserver with Master: 917d396fa2334e068c36f5e3797899d2 (127.0.98.1:35345)
I20260925 14:50:44.283553 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47227
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:44.305984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:44.387005 3652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:44.387213 3652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:44.387238 3652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:44.388438 3652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:44.388481 3652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:44.388540 3652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:44.389985 3652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:44.390268 3652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:44.390532 3652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:44.395671 3660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.395677 3658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.395720 3657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:44.395994 3652 server_base.cc:1034] running on GCE node
I20260925 14:50:44.396255 3652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:44.396507 3652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:44.397661 3652 hybrid_clock.cc:648] HybridClock initialized: now 1790347844397636 us; error 46 us; skew 500 ppm
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:44.400247 3652 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:44.400720 3652 webserver.cc:460] Webserver started at http://127.0.98.2:43545/ using document root <none> and password file <none>
I20260925 14:50:44.400956 3652 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:44.401000 3652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:44.401146 3652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:44.402186 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "092b2d45b46a458e9d5f8bd1eb941ff4"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "6de4166c24c8755b610587a838e36172"
server_key_iv: "244eec3b70b586e34af1db3c485a2acc"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.402633 3652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "092b2d45b46a458e9d5f8bd1eb941ff4"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "6de4166c24c8755b610587a838e36172"
server_key_iv: "244eec3b70b586e34af1db3c485a2acc"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.404093 3652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:44.404845 3667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.405027 3652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:44.405117 3652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "092b2d45b46a458e9d5f8bd1eb941ff4"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "6de4166c24c8755b610587a838e36172"
server_key_iv: "244eec3b70b586e34af1db3c485a2acc"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.405192 3652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:44.430979 3652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:44.433492 3652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:44.433665 3652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:44.433917 3652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:44.434255 3652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:44.434309 3652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.434340 3652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:44.434384 3652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.440434 3652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37893
I20260925 14:50:44.440488 3780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37893 every 8 connection(s)
I20260925 14:50:44.440817 3652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:44.445099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3652
I20260925 14:50:44.445228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:44.445426 392 external_mini_cluster.cc:1428] Setting key 47ce3c460ee25f714b2fad8212c94b58
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:44.450502 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.441533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43661 (local address 127.0.98.62:35749)
0925 14:50:44.441597 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:44.441599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:44.441812 (+ 213us) server_negotiation.cc:407] Connection header received
0925 14:50:44.442281 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:44.442282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:44.442291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:44.442436 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:44.443388 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.443900 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.444892 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.445097 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.445210 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.447612 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:44.447614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:44.447615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:44.447623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:44.449057 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.449494 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.449496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.449496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.449557 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.449835 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.449836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.449837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.449952 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:44.450028 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:44.450297 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:50:44.450414 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:50:44.450685 3783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.441499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35749 (local address 127.0.98.2:43661)
0925 14:50:44.441677 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:44.441719 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:44.442199 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:44.442517 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:44.442520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:44.443072 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:44.443263 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.443267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.443986 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.443987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.444799 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.444801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.445221 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.445222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.445385 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.445939 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:44.445948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:44.447511 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:44.449166 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.449173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.449175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.449408 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.449633 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.449633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.449634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.449752 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.450019 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:44.450026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:44.450168 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 14:50:44.450276 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:50:44.450453 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260925 14:50:44.451161 3781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35749
I20260925 14:50:44.451273 3781 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:44.451483 3781 heartbeater.cc:511] Master 127.0.98.62:35749 requested a full tablet report, sending...
I20260925 14:50:44.451834 3456 ts_manager.cc:194] Registered new tserver with Master: 092b2d45b46a458e9d5f8bd1eb941ff4 (127.0.98.2:37893)
I20260925 14:50:44.452203 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43661
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:44.469583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:44.551862 3786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:44.552071 3786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:44.552095 3786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:44.553267 3786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:44.553314 3786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:44.553341 3786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:44.554848 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:44.555095 3786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:44.555332 3786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:44.560395 3794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.560401 3791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:44.560451 3792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:44.560658 3786 server_base.cc:1034] running on GCE node
I20260925 14:50:44.560856 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:44.561054 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:44.562188 3786 hybrid_clock.cc:648] HybridClock initialized: now 1790347844562170 us; error 32 us; skew 500 ppm
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:44.564746 3786 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:44.565171 3786 webserver.cc:460] Webserver started at http://127.0.98.3:36631/ using document root <none> and password file <none>
I20260925 14:50:44.565405 3786 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:44.565450 3786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:44.565589 3786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:44.566710 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "d5155235d0114ea8a0c3ee170671f390"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "1cba3a270b9b924918e25f913720454e"
server_key_iv: "fc1d5f0df9dd9721fc18bd723a0e9400"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.567075 3786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d5155235d0114ea8a0c3ee170671f390"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "1cba3a270b9b924918e25f913720454e"
server_key_iv: "fc1d5f0df9dd9721fc18bd723a0e9400"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.568420 3786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:50:44.569156 3801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.569370 3786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:44.569461 3786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "d5155235d0114ea8a0c3ee170671f390"
format_stamp: "Formatted at 2026-09-25 14:50:44 on dist-test-slave-gpc1"
server_key: "1cba3a270b9b924918e25f913720454e"
server_key_iv: "fc1d5f0df9dd9721fc18bd723a0e9400"
server_key_version: "encryptionkey@0"
I20260925 14:50:44.569535 3786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:44.602547 3786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:44.605165 3786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:44.605369 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:44.605599 3786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:44.605919 3786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:44.605974 3786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.606016 3786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:44.606045 3786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:44.611948 3786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43893
I20260925 14:50:44.612002 3914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43893 every 8 connection(s)
I20260925 14:50:44.612341 3786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:44.615384 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3786
I20260925 14:50:44.615473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:44.615684 392 external_mini_cluster.cc:1428] Setting key 3690100d21b1b86332c875bb1d0a6f64
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:44.621636 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.613023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49857 (local address 127.0.98.62:35749)
0925 14:50:44.613064 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:44.613066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:44.613317 (+ 251us) server_negotiation.cc:407] Connection header received
0925 14:50:44.613811 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:44.613812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:44.613822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:44.613876 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:44.614891 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.615558 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.616914 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.617093 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.617147 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.619245 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:44.619246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:44.619246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:44.619252 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:44.620464 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.620819 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.620819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.620820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.620878 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.621117 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.621117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.621117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.621211 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:44.621259 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:44.621518 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:50:44.621575 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:44.621863 3917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.613004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35749 (local address 127.0.98.3:49857)
0925 14:50:44.613146 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:44.613180 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:44.613702 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:44.613933 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:44.613937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:44.614568 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:44.614771 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.614777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.615654 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.615655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.616851 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:44.616854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.617163 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.617163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:44.617261 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.617835 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:44.617844 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:44.619169 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:44.620544 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.620549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.620551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.620756 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.620953 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:44.620954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:44.620954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:44.621058 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:44.621274 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:44.621277 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:44.621413 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:50:44.621495 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:50:44.621607 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260925 14:50:44.622323 3915 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35749
I20260925 14:50:44.622455 3915 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:44.622618 3915 heartbeater.cc:511] Master 127.0.98.62:35749 requested a full tablet report, sending...
I20260925 14:50:44.623006 3456 ts_manager.cc:194] Registered new tserver with Master: d5155235d0114ea8a0c3ee170671f390 (127.0.98.3:43893)
I20260925 14:50:44.623337 3456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49857
I20260925 14:50:44.627267 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347844, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:44.649065 3496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:44.641998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.0.98.62:35749)
0925 14:50:44.642059 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:44.642065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:44.642173 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:50:44.642234 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:44.642235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:44.642244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:44.642306 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:44.642916 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.643409 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.644152 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:44.644346 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:44.644400 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:44.646685 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:44.646687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:44.646687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:44.646693 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:44.648134 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.648435 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.648435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.648436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.648483 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:44.648637 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:44.648637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:44.648637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:44.648766 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:44.648854 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:44.648938 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:50:44.648991 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 14:50:44.649308 3456 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50552
I20260925 14:50:44.649835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3518
I20260925 14:50:44.654276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3652
I20260925 14:50:44.658811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3786
I20260925 14:50:44.662938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3424
2026-09-25T14:50:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1856 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3927](info): set up 2 sockets
Sep 25 14:50:44 dist-test-slave-gpc1 krb5kdc[3927](info): commencing operation
krb5kdc: starting...
W20260925 14:50:45.690258 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:45 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:45.810011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:45.892189 3940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:45.892421 3940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:45.892444 3940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:45.893646 3940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:45.893690 3940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:45.893705 3940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:45.893718 3940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:45.893729 3940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:45.895231 3940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:45.895505 3940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:45.895737 3940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:45.900848 3948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:45.900815 3945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:45.900869 3946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:45.901000 3940 server_base.cc:1034] running on GCE node
I20260925 14:50:45.901330 3940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:45.901583 3940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:45.902725 3940 hybrid_clock.cc:648] HybridClock initialized: now 1790347845902692 us; error 46 us; skew 500 ppm
Sep 25 14:50:45 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:45.905049 3940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:45.905534 3940 webserver.cc:460] Webserver started at http://127.0.98.62:43745/ using document root <none> and password file <none>
I20260925 14:50:45.905728 3940 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:45.905769 3940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:45.905855 3940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:45.906916 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "6e10c79a47e64dd78e07ec032e9b53cd"
format_stamp: "Formatted at 2026-09-25 14:50:45 on dist-test-slave-gpc1"
server_key: "fc6e9994e19c56fd39b00d37c8d3c17e"
server_key_iv: "5ab7c669e9069c364f19006517df1742"
server_key_version: "encryptionkey@0"
I20260925 14:50:45.907217 3940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e10c79a47e64dd78e07ec032e9b53cd"
format_stamp: "Formatted at 2026-09-25 14:50:45 on dist-test-slave-gpc1"
server_key: "fc6e9994e19c56fd39b00d37c8d3c17e"
server_key_iv: "5ab7c669e9069c364f19006517df1742"
server_key_version: "encryptionkey@0"
I20260925 14:50:45.908569 3940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:45.909308 3955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:45.909533 3940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:45.909637 3940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "6e10c79a47e64dd78e07ec032e9b53cd"
format_stamp: "Formatted at 2026-09-25 14:50:45 on dist-test-slave-gpc1"
server_key: "fc6e9994e19c56fd39b00d37c8d3c17e"
server_key_iv: "5ab7c669e9069c364f19006517df1742"
server_key_version: "encryptionkey@0"
I20260925 14:50:45.909716 3940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:45.916486 3940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:45.918628 3940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:45.918780 3940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:45.922770 3940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33821
I20260925 14:50:45.922811 4007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33821 every 8 connection(s)
I20260925 14:50:45.923262 3940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:45.924675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 3940
I20260925 14:50:45.924798 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:45.924868 4008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:45.925000 392 external_mini_cluster.cc:1428] Setting key d644b3becbb67cd7139a271de2f9eb54
I20260925 14:50:45.928479 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: Bootstrap starting.
I20260925 14:50:45.929085 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:45.929401 4008 log.cc:824] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:45.930154 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: No bootstrap required, opened a new log
Sep 25 14:50:45 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:45.932113 4008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } }
I20260925 14:50:45.932286 4008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:45.932333 4008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e10c79a47e64dd78e07ec032e9b53cd, State: Initialized, Role: FOLLOWER
I20260925 14:50:45.932482 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } }
I20260925 14:50:45.932574 4008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:45.932613 4008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:45.932664 4008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:45.933372 4008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } }
I20260925 14:50:45.933501 4008 leader_election.cc:302] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e10c79a47e64dd78e07ec032e9b53cd; no voters:
I20260925 14:50:45.933682 4008 leader_election.cc:288] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:45.933760 4013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:45.933938 4013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [term 1 LEADER]: Becoming Leader. State: Replica: 6e10c79a47e64dd78e07ec032e9b53cd, State: Running, Role: LEADER
I20260925 14:50:45.933954 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:45.934041 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } }
I20260925 14:50:45.934417 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e10c79a47e64dd78e07ec032e9b53cd. Latest consensus state: current_term: 1 leader_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } } }
I20260925 14:50:45.934506 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:45.935739 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e10c79a47e64dd78e07ec032e9b53cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33821 } } }
I20260925 14:50:45.935956 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd [sys.catalog]: This master's current role is: LEADER
W20260925 14:50:45.936195 4028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:45.936429 4028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:45.937088 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:45.925533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.0.98.62:33821)
0925 14:50:45.925697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:45.925706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:45.925739 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:50:45.926321 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:45.926330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:45.926406 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:45.926557 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:45.927163 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:45.928494 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:45.929390 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:45.929962 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:45.930072 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:45.932438 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:45.932442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:45.932443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:45.932457 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:45.934766 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:45.935262 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:45.935265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:45.935266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:45.935387 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:45.935646 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:45.935647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:45.935647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:45.936038 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:45.936675 (+ 637us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:45.936748 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 14:50:45.936887 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 14:50:45.937265 4029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:45.937444 4029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:45.938974 4029 catalog_manager.cc:1347] Generated new cluster ID: 0d6ff45a17d244dbb81dfc242fd3debe
I20260925 14:50:45.939033 4029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:45.954144 4029 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:45.954977 4029 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:45.963114 4029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e10c79a47e64dd78e07ec032e9b53cd: Generated new TSK 0
I20260925 14:50:45.963317 4029 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:45.993815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:46.075577 4034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:46.075771 4034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:46.075794 4034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:46.076941 4034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:46.076979 4034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:46.077006 4034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:46.078540 4034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:46.078779 4034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:46.079010 4034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:46.083838 4039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:46.083928 4042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:46.083890 4040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:46.083995 4034 server_base.cc:1034] running on GCE node
I20260925 14:50:46.084221 4034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:46.084530 4034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:46.085695 4034 hybrid_clock.cc:648] HybridClock initialized: now 1790347846085684 us; error 28 us; skew 500 ppm
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:46.088299 4034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:46.088754 4034 webserver.cc:460] Webserver started at http://127.0.98.1:37781/ using document root <none> and password file <none>
I20260925 14:50:46.089003 4034 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:46.089048 4034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:46.089210 4034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:46.090265 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed256d2281a342dc8123e0001b54dd3a"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "5021d00313b4b2707539644082bc6ee7"
server_key_iv: "70c74c0b04d1aeb45c334a46bc95d2c8"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.090687 4034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed256d2281a342dc8123e0001b54dd3a"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "5021d00313b4b2707539644082bc6ee7"
server_key_iv: "70c74c0b04d1aeb45c334a46bc95d2c8"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.091971 4034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:46.092679 4049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.092845 4034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:46.092918 4034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "ed256d2281a342dc8123e0001b54dd3a"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "5021d00313b4b2707539644082bc6ee7"
server_key_iv: "70c74c0b04d1aeb45c334a46bc95d2c8"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.092985 4034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:46.110587 4034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:46.112525 4034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:46.112636 4034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:46.112836 4034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:46.113132 4034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:46.113168 4034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.113191 4034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:46.113204 4034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.118814 4034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40019
I20260925 14:50:46.118901 4162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40019 every 8 connection(s)
I20260925 14:50:46.119210 4034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:46.128571 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.119940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39835 (local address 127.0.98.62:33821)
0925 14:50:46.120008 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:46.120010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:46.120224 (+ 214us) server_negotiation.cc:407] Connection header received
0925 14:50:46.121205 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:46.121206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:46.121221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:46.121263 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:46.121899 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.122483 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.123492 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.123708 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.123795 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.125818 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:46.125819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:46.125820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:46.125827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:46.127142 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.127588 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.127589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.127589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.127649 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.127900 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.127901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.127901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.128013 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:46.128083 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:46.128373 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 14:50:46.128476 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:50:46.128772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4034
I20260925 14:50:46.128804 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.119940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33821 (local address 127.0.98.1:39835)
0925 14:50:46.120098 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:46.120140 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:46.121081 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:46.121352 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:46.121358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:46.121613 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:46.121799 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.121804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.122571 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.122572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.123395 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.123397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.123804 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.123805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.123940 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.124397 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:46.124405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:46.125709 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:46.127234 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.127241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.127243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.127492 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.127715 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.127716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.127717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.127823 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.128085 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:46.128092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:46.128227 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:50:46.128347 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:50:46.128498 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260925 14:50:46.128895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:46.129094 392 external_mini_cluster.cc:1428] Setting key 7a0bfa29399e985a5f134e6aa89644cd
I20260925 14:50:46.129463 4163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33821
I20260925 14:50:46.129555 4163 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:46.129776 4163 heartbeater.cc:511] Master 127.0.98.62:33821 requested a full tablet report, sending...
I20260925 14:50:46.130302 3972 ts_manager.cc:194] Registered new tserver with Master: ed256d2281a342dc8123e0001b54dd3a (127.0.98.1:40019)
I20260925 14:50:46.130978 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39835
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:46.153663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:46.234665 4168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:46.234860 4168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:46.234882 4168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:46.236097 4168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:46.236171 4168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:46.236202 4168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:46.237597 4168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:46.237829 4168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:46.238014 4168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:46.242981 4174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:46.243081 4168 server_base.cc:1034] running on GCE node
W20260925 14:50:46.242972 4176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:46.242972 4173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:46.243431 4168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:46.243647 4168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:46.244783 4168 hybrid_clock.cc:648] HybridClock initialized: now 1790347846244766 us; error 34 us; skew 500 ppm
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:46.247423 4168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:46.247895 4168 webserver.cc:460] Webserver started at http://127.0.98.2:46151/ using document root <none> and password file <none>
I20260925 14:50:46.248113 4168 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:46.248154 4168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:46.248242 4168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:46.249233 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "4dc96eb131514fb69ac5a4a8813dd230"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "9a155dd91a41c1c472daf53ea52fb829"
server_key_iv: "6cee6cf52e16c6260a638f7f74b8e79c"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.249537 4168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4dc96eb131514fb69ac5a4a8813dd230"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "9a155dd91a41c1c472daf53ea52fb829"
server_key_iv: "6cee6cf52e16c6260a638f7f74b8e79c"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.250862 4168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:50:46.251564 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.251739 4168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:46.251829 4168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "4dc96eb131514fb69ac5a4a8813dd230"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "9a155dd91a41c1c472daf53ea52fb829"
server_key_iv: "6cee6cf52e16c6260a638f7f74b8e79c"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.251910 4168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:46.261010 4168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:46.263376 4168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:46.263540 4168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:46.263772 4168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:46.264070 4168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:46.264122 4168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.264163 4168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:46.264191 4168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.269918 4168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37969
I20260925 14:50:46.269969 4296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37969 every 8 connection(s)
I20260925 14:50:46.270299 4168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:46.278501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4168
I20260925 14:50:46.278635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:46.278839 392 external_mini_cluster.cc:1428] Setting key b03f77f3306bebee58f0df148f059203
I20260925 14:50:46.280078 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.271083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39161 (local address 127.0.98.62:33821)
0925 14:50:46.271165 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:46.271167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:46.271408 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:50:46.272380 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:46.272381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:46.272394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:46.272447 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:46.273072 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.273585 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.274679 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.274879 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.274972 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.277017 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:46.277018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:46.277018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:46.277025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:46.278498 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.279006 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.279007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.279008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.279091 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.279364 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.279365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.279365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.279512 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:46.279586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:46.279944 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 14:50:46.280015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:50:46.280380 4299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.271046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33821 (local address 127.0.98.2:39161)
0925 14:50:46.271221 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:46.271286 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:46.272257 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:46.272518 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:46.272522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:46.272778 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:46.272978 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.272982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.273683 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.273684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.274580 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.274583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.274991 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.274992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.275112 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.275588 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:46.275595 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:46.276924 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:46.278576 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.278583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.278586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.278894 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.279164 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.279165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.279166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.279297 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.279600 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:46.279606 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:46.279780 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:50:46.279922 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 14:50:46.280105 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260925 14:50:46.280895 4297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33821
I20260925 14:50:46.281018 4297 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:46.281200 4297 heartbeater.cc:511] Master 127.0.98.62:33821 requested a full tablet report, sending...
I20260925 14:50:46.281623 3972 ts_manager.cc:194] Registered new tserver with Master: 4dc96eb131514fb69ac5a4a8813dd230 (127.0.98.2:37969)
I20260925 14:50:46.282061 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39161
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:46.303649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:46.384522 4302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:46.384752 4302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:46.384776 4302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:46.385902 4302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:46.385939 4302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:46.385965 4302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:46.387375 4302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:46.387562 4302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:46.387758 4302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:46.392884 4310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:46.392894 4308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:46.392925 4302 server_base.cc:1034] running on GCE node
W20260925 14:50:46.392951 4307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:46.393361 4302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:46.393630 4302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:46.394805 4302 hybrid_clock.cc:648] HybridClock initialized: now 1790347846394790 us; error 30 us; skew 500 ppm
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:46.397291 4302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:46.397729 4302 webserver.cc:460] Webserver started at http://127.0.98.3:35535/ using document root <none> and password file <none>
I20260925 14:50:46.397969 4302 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:46.398015 4302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:46.398149 4302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:46.399258 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "b64a5b04fa264e498ca6353ae785c6ef"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "03211f2ab8bb882d27dd37b680e4fb87"
server_key_iv: "3fa4d69f8101a94bbd9ed2d9536d66cc"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.399613 4302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b64a5b04fa264e498ca6353ae785c6ef"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "03211f2ab8bb882d27dd37b680e4fb87"
server_key_iv: "3fa4d69f8101a94bbd9ed2d9536d66cc"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.400936 4302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:46.401655 4317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.401839 4302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:46.401937 4302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "b64a5b04fa264e498ca6353ae785c6ef"
format_stamp: "Formatted at 2026-09-25 14:50:46 on dist-test-slave-gpc1"
server_key: "03211f2ab8bb882d27dd37b680e4fb87"
server_key_iv: "3fa4d69f8101a94bbd9ed2d9536d66cc"
server_key_version: "encryptionkey@0"
I20260925 14:50:46.402017 4302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:46.412185 4302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:46.414449 4302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:46.414600 4302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:46.414835 4302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:46.415133 4302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:46.415185 4302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.415226 4302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:46.415253 4302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:46.421242 4302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41311
I20260925 14:50:46.421278 4430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41311 every 8 connection(s)
I20260925 14:50:46.421622 4302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:46.428486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4302
I20260925 14:50:46.428583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:46.428759 392 external_mini_cluster.cc:1428] Setting key 290b35009291a2070df71d9caaced1ad
I20260925 14:50:46.431986 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.422534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60711 (local address 127.0.98.62:33821)
0925 14:50:46.422603 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:46.422605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:46.422814 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:50:46.424087 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:46.424089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:46.424103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:46.424136 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:46.425033 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.425536 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.426593 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.426779 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.426839 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.429099 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:46.429100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:46.429101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:46.429108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:46.430667 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.431017 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.431018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.431018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.431070 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.431272 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.431272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.431273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.431390 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:46.431458 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:46.431774 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 14:50:46.431857 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:50:46.432283 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.422509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33821 (local address 127.0.98.3:60711)
0925 14:50:46.422662 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:46.422703 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:46.423949 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:46.424324 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:46.424329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:46.424732 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:46.424938 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.424942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.425628 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.425629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.426519 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:46.426521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.426853 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.426854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:46.426954 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.427425 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:46.427433 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:46.429017 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:46.430734 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.430740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.430742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.430954 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.431126 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:46.431127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:46.431127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:46.431220 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:46.431447 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:46.431471 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:46.431631 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 14:50:46.431749 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:50:46.431926 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260925 14:50:46.432948 4431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33821
I20260925 14:50:46.433043 4431 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:46.433187 4431 heartbeater.cc:511] Master 127.0.98.62:33821 requested a full tablet report, sending...
I20260925 14:50:46.433594 3972 ts_manager.cc:194] Registered new tserver with Master: b64a5b04fa264e498ca6353ae785c6ef (127.0.98.3:41311)
I20260925 14:50:46.433990 3972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60711
I20260925 14:50:46.440667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:46.462809 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.455386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.0.98.1:40019)
0925 14:50:46.455561 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:46.455567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:46.455586 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:50:46.455652 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:46.455655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:46.455672 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:46.455752 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:46.456323 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.457039 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.457813 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.458126 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.458179 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.460377 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:46.460385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:46.460385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:46.460391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:46.461773 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.462082 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.462085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.462085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.462138 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.462294 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.462295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.462295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.462470 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:46.462581 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:46.462641 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:50:46.462698 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[3927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:46.471271 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:46.464142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.98.62:33821)
0925 14:50:46.464207 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:46.464209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:46.464219 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:46.464281 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:46.464281 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:46.464292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:46.464381 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:46.464835 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.465333 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.466012 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:46.466220 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:46.466305 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:46.468545 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:46.468546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:46.468547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:46.468553 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:46.469951 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.470426 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.470428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.470428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.470491 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:46.470710 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:46.470711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:46.470711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:46.470867 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:46.470973 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:46.471077 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:50:46.471162 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 14:50:46.471520 3972 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50350
I20260925 14:50:46.471873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4034
I20260925 14:50:46.478003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4168
I20260925 14:50:46.483232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4302
I20260925 14:50:46.487286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 3940
2026-09-25T14:50:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1824 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[4443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[4443](info): set up 2 sockets
Sep 25 14:50:46 dist-test-slave-gpc1 krb5kdc[4443](info): commencing operation
krb5kdc: starting...
W20260925 14:50:47.522202 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:47 dist-test-slave-gpc1 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:47.633116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:47.714728 4456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:47.714936 4456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:47.714962 4456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:47.716281 4456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:47.716326 4456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:47.716342 4456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:47.716355 4456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:47.716370 4456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:47.718055 4456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:47.718323 4456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:47.718613 4456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:47.723757 4462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:47.723918 4456 server_base.cc:1034] running on GCE node
W20260925 14:50:47.723847 4464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:47.723767 4461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:47.724210 4456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:47.724449 4456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:47.725615 4456 hybrid_clock.cc:648] HybridClock initialized: now 1790347847725597 us; error 32 us; skew 500 ppm
Sep 25 14:50:47 dist-test-slave-gpc1 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:47.728108 4456 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:47.728384 4456 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:47.728425 4456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:47.728511 4456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:47.729550 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e19cf948f7124187bc6748967588c4bf"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "dd3a1027dec67aa31a74113c056ea537"
server_key_iv: "f777b5b9fd4708747af55a0adaff4fa3"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.729853 4456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e19cf948f7124187bc6748967588c4bf"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "dd3a1027dec67aa31a74113c056ea537"
server_key_iv: "f777b5b9fd4708747af55a0adaff4fa3"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.731163 4456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:47.731889 4470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:47.732065 4456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:47.732148 4456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e19cf948f7124187bc6748967588c4bf"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "dd3a1027dec67aa31a74113c056ea537"
server_key_iv: "f777b5b9fd4708747af55a0adaff4fa3"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.732201 4456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:47.741495 4456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:47.743428 4456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:47.743536 4456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:47.747189 4456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45029
I20260925 14:50:47.747287 4522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45029 every 8 connection(s)
I20260925 14:50:47.747498 4456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:47.748435 4523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:47.750854 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: Bootstrap starting.
I20260925 14:50:47.751435 4523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:47.751724 4523 log.cc:824] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:47.752441 4523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: No bootstrap required, opened a new log
I20260925 14:50:47.753966 4523 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } }
I20260925 14:50:47.754132 4523 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:47.754181 4523 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19cf948f7124187bc6748967588c4bf, State: Initialized, Role: FOLLOWER
I20260925 14:50:47.754343 4523 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } }
I20260925 14:50:47.754462 4523 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:47.754506 4523 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:47.754554 4523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:47.755317 4523 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } }
I20260925 14:50:47.755432 4523 leader_election.cc:302] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19cf948f7124187bc6748967588c4bf; no voters:
I20260925 14:50:47.755613 4523 leader_election.cc:288] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:47.755662 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:47.755844 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [term 1 LEADER]: Becoming Leader. State: Replica: e19cf948f7124187bc6748967588c4bf, State: Running, Role: LEADER
I20260925 14:50:47.755867 4523 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:47.755932 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } }
I20260925 14:50:47.756328 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19cf948f7124187bc6748967588c4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } } }
I20260925 14:50:47.756364 4528 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19cf948f7124187bc6748967588c4bf. Latest consensus state: current_term: 1 leader_uuid: "e19cf948f7124187bc6748967588c4bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19cf948f7124187bc6748967588c4bf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45029 } } }
I20260925 14:50:47.756438 4528 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:47.756428 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:47.757047 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:47.757493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4456
I20260925 14:50:47.757552 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:47.757602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:47.757803 392 external_mini_cluster.cc:1428] Setting key f7103a0df4ec5089305e3b162f448f1d
I20260925 14:50:47.759140 4536 catalog_manager.cc:1347] Generated new cluster ID: 93c3bd9399d243ddb9ea6ab39da80e2e
I20260925 14:50:47.759191 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:47 dist-test-slave-gpc1 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:47.768174 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:47.768778 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:47.769598 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:47.758233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.98.62:45029)
0925 14:50:47.758420 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:47.758430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:47.758503 (+ 73us) server_negotiation.cc:407] Connection header received
0925 14:50:47.758658 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:47.758664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:47.758754 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:47.758915 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:47.759657 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.761050 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.762117 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.762748 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.762849 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:47.765663 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:47.765666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:47.765667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:47.765679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:47.767842 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:47.768292 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:47.768295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:47.768295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:47.768405 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:47.768621 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:47.768622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:47.768622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:47.768948 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:47.769124 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:47.769220 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:50:47.769366 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260925 14:50:47.787748 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: Generated new TSK 0
I20260925 14:50:47.787959 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:47.815513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:47.899739 4549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:47.899967 4549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:47.900007 4549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:47.901154 4549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:47.901217 4549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:47.901263 4549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:47.902817 4549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:47.903066 4549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:47.903283 4549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:47.908089 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:47.908041 4557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:47.908046 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:47.908375 4549 server_base.cc:1034] running on GCE node
I20260925 14:50:47.908612 4549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:47.908835 4549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:47.909973 4549 hybrid_clock.cc:648] HybridClock initialized: now 1790347847909979 us; error 42 us; skew 500 ppm
Sep 25 14:50:47 dist-test-slave-gpc1 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:47.912600 4549 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:47.912882 4549 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:47.912928 4549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:47.913019 4549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:47.914049 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "bbb5698ff1ee415ca0dd8b8971564eb1"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "cfcca373d00a26822042bd65541bb9d0"
server_key_iv: "4158a6e560120978c2c6973ee4c00247"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.914449 4549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bbb5698ff1ee415ca0dd8b8971564eb1"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "cfcca373d00a26822042bd65541bb9d0"
server_key_iv: "4158a6e560120978c2c6973ee4c00247"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.915738 4549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:47.916424 4563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:47.916574 4549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:47.916661 4549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "bbb5698ff1ee415ca0dd8b8971564eb1"
format_stamp: "Formatted at 2026-09-25 14:50:47 on dist-test-slave-gpc1"
server_key: "cfcca373d00a26822042bd65541bb9d0"
server_key_iv: "4158a6e560120978c2c6973ee4c00247"
server_key_version: "encryptionkey@0"
I20260925 14:50:47.916711 4549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:47.928627 4549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:47.930634 4549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:47.930742 4549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:47.930907 4549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:47.931211 4549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:47.931245 4549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:47.931267 4549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:47.931279 4549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:47.937089 4549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40935
I20260925 14:50:47.937170 4676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40935 every 8 connection(s)
I20260925 14:50:47.937433 4549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:47.940325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4549
I20260925 14:50:47.940428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:47.940595 392 external_mini_cluster.cc:1428] Setting key e5e68959fa200ca80a68974f7e3193fa
Sep 25 14:50:47 dist-test-slave-gpc1 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:47.947665 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:47.938133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46621 (local address 127.0.98.62:45029)
0925 14:50:47.938206 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:47.938209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:47.938462 (+ 253us) server_negotiation.cc:407] Connection header received
0925 14:50:47.938946 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:47.938948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:47.938960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:47.939006 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:47.939896 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.940583 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.941799 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.942144 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.942218 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:47.944755 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:47.944757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:47.944757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:47.944764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:47.946218 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:47.946680 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:47.946681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:47.946682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:47.946742 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:47.947024 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:47.947025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:47.947026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:47.947137 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:47.947214 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:47.947488 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 14:50:47.947576 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:50:47.947844 4679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:47.938129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45029 (local address 127.0.98.1:46621)
0925 14:50:47.938315 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:47.938381 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:47.938859 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:47.939067 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:47.939070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:47.939314 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:47.939796 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:47.939801 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.940699 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.940700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:47.941706 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:47.941708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:47.942233 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:47.942233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:47.942395 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:47.943039 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:47.943048 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:47.944659 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:47.946323 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:47.946330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:47.946332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:47.946585 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:47.946827 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:47.946828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:47.946828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:47.946950 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:47.947204 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:47.947210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:47.947353 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:50:47.947455 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:50:47.947611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260925 14:50:47.948297 4677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45029
I20260925 14:50:47.948406 4677 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:47.948546 4677 heartbeater.cc:511] Master 127.0.98.62:45029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:50:47.949071 4487 ts_manager.cc:194] Registered new tserver with Master: bbb5698ff1ee415ca0dd8b8971564eb1 (127.0.98.1:40935)
I20260925 14:50:47.949734 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:47.965121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:48.044399 4682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:48.044603 4682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:48.044627 4682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:48.045796 4682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:48.045840 4682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:48.045866 4682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:48.047367 4682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:48.047598 4682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:48.047821 4682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:48.053156 4688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.053093 4687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.053328 4682 server_base.cc:1034] running on GCE node
W20260925 14:50:48.053092 4690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.053599 4682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:48.053854 4682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:48.055018 4682 hybrid_clock.cc:648] HybridClock initialized: now 1790347848054998 us; error 34 us; skew 500 ppm
Sep 25 14:50:48 dist-test-slave-gpc1 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:48.057794 4682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:48.058125 4682 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:48.058172 4682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:48.058396 4682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:48.059468 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "13f58c48cdf14064b8b0f9fc49f406e1"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "0838719c7d86e773d797fb4ace7cf643"
server_key_iv: "3d961a72e26ce342d2b536c0f6f1b5df"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.059815 4682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "13f58c48cdf14064b8b0f9fc49f406e1"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "0838719c7d86e773d797fb4ace7cf643"
server_key_iv: "3d961a72e26ce342d2b536c0f6f1b5df"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.061072 4682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:48.061812 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.061985 4682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:48.062069 4682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "13f58c48cdf14064b8b0f9fc49f406e1"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "0838719c7d86e773d797fb4ace7cf643"
server_key_iv: "3d961a72e26ce342d2b536c0f6f1b5df"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.062121 4682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:48.078184 4682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:48.080142 4682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:48.080257 4682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:48.080441 4682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:48.080694 4682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:48.080724 4682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.080745 4682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:48.080757 4682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.086076 4682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42469
I20260925 14:50:48.086182 4809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42469 every 8 connection(s)
I20260925 14:50:48.086364 4682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:48.089532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4682
I20260925 14:50:48.089658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:48.089854 392 external_mini_cluster.cc:1428] Setting key 22125bb657accd59fdbdd160e456dc69
Sep 25 14:50:48 dist-test-slave-gpc1 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:48.096783 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.087065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49505 (local address 127.0.98.62:45029)
0925 14:50:48.087107 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.087109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.087282 (+ 173us) server_negotiation.cc:407] Connection header received
0925 14:50:48.087797 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.087798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.087811 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.087858 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:48.088784 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.089290 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.090670 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.090878 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.090960 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.093590 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:48.093591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:48.093592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:48.093598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:48.095078 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.095592 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.095593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.095593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.095642 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.095929 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.095929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.095929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.096070 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:48.096153 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.096533 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.096670 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:48.096899 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.087019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45029 (local address 127.0.98.2:49505)
0925 14:50:48.087160 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.087202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.087689 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.087953 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.087956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.088202 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:48.088676 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.088681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.089419 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.089420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.090581 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.090583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.090969 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.090971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.091107 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.091650 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:48.091663 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:48.093500 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:48.095164 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:48.095170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:48.095173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:48.095516 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:48.095710 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:48.095711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:48.095711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:48.095862 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:48.096161 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:48.096168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:48.096363 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.096481 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.096646 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260925 14:50:48.097724 4810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45029
I20260925 14:50:48.097832 4810 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:48.097988 4810 heartbeater.cc:511] Master 127.0.98.62:45029 requested a full tablet report, sending...
I20260925 14:50:48.098435 4487 ts_manager.cc:194] Registered new tserver with Master: 13f58c48cdf14064b8b0f9fc49f406e1 (127.0.98.2:42469)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:50:48.098965 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:48.116616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:48.201092 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:48.201346 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:48.201390 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:48.202600 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:48.202672 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:48.202719 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:48.204244 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:48.204567 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:48.204804 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:48.209918 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.209975 4815 server_base.cc:1034] running on GCE node
W20260925 14:50:48.209897 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.209892 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.210467 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:48.210747 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:48.211918 4815 hybrid_clock.cc:648] HybridClock initialized: now 1790347848211900 us; error 36 us; skew 500 ppm
Sep 25 14:50:48 dist-test-slave-gpc1 krb5kdc[4443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:48.222664 4815 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:48.223064 4815 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:48.223136 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:48.223295 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:48.224382 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "14d2eddaa43945f2b2e307fe70329710"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "e8424fff8470f25a75d35acdd090b794"
server_key_iv: "571aa1fc045e6706d0ed88ad523839b3"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.224756 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "14d2eddaa43945f2b2e307fe70329710"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "e8424fff8470f25a75d35acdd090b794"
server_key_iv: "571aa1fc045e6706d0ed88ad523839b3"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.226132 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:48.227018 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.227216 4815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:48.227313 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "14d2eddaa43945f2b2e307fe70329710"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "e8424fff8470f25a75d35acdd090b794"
server_key_iv: "571aa1fc045e6706d0ed88ad523839b3"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.227422 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:48.242130 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:48.244478 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:48.244649 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:48.244884 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:48.245189 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:48.245242 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.245282 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:48.245311 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.251331 4815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36495
I20260925 14:50:48.251367 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36495 every 8 connection(s)
I20260925 14:50:48.251631 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:50:48 dist-test-slave-gpc1 krb5kdc[4443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:48.260804 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.252330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60571 (local address 127.0.98.62:45029)
0925 14:50:48.252415 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.252417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.252599 (+ 182us) server_negotiation.cc:407] Connection header received
0925 14:50:48.253062 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.253063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.253072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.253129 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:48.254054 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.254631 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.255688 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.255876 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.255930 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.258268 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:48.258270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:48.258270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:48.258293 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:48.259522 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.259944 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.259945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.259945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.259993 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.260213 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.260214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.260214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.260342 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:48.260396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.260678 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.260741 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:50:48.261049 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.252297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45029 (local address 127.0.98.3:60571)
0925 14:50:48.252469 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.252512 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.252981 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.253183 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.253186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.253474 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:48.253965 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.253970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.254707 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.254708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.255611 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.255614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.255945 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.255946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.256060 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.256576 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:48.256585 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:48.258183 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:48.259615 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:48.259621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:48.259622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:48.259872 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:48.260050 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:48.260050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:48.260051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:48.260151 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:48.260434 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:48.260439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:48.260557 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.260653 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.260771 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":108,"threads_started":1}
I20260925 14:50:48.261572 4943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45029
I20260925 14:50:48.261617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4815
I20260925 14:50:48.261685 4943 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:48.261718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:48.261852 4943 heartbeater.cc:511] Master 127.0.98.62:45029 requested a full tablet report, sending...
I20260925 14:50:48.261945 392 external_mini_cluster.cc:1428] Setting key c26865d5ae5ad8705ff970e7faba9dbe
I20260925 14:50:48.262277 4487 ts_manager.cc:194] Registered new tserver with Master: 14d2eddaa43945f2b2e307fe70329710 (127.0.98.3:36495)
I20260925 14:50:48.262746 4487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60571
I20260925 14:50:48.262758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:48.270226 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.264242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.0.98.62:45029)
0925 14:50:48.264293 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.264294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.264424 (+ 130us) server_negotiation.cc:407] Connection header received
0925 14:50:48.264515 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.264516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.264526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.264670 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:48.265294 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.265762 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.266603 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.266805 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.266888 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.267993 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:48.267994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:48.267994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:48.268001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:48.269122 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.269462 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.269463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.269463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.269522 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:48.269763 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:48.269764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:48.269764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:48.269887 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:48.269966 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.270113 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.270168 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:50:48.272226 4487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:48.273875 4487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:48.280356 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42469 (local address 127.0.0.1:34234)
0925 14:50:48.276620 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.276630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.276668 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.276987 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.276988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.276991 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:48.277043 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.277045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.278336 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.278364 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.279340 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.279343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.280063 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.280140 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280154 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.280220 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.280270 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":35,"threads_started":1}
I20260925 14:50:48.280471 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.0.98.3:36495)
0925 14:50:48.276581 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.276587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.276598 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:50:48.276637 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.276640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.276657 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.277059 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:48.277131 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.277860 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.279253 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.280067 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280175 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280231 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.280324 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.280368 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":90,"threads_started":1}
I20260925 14:50:48.280668 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.0.98.2:42469)
0925 14:50:48.276724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.276729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.276739 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:48.276781 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.276783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.276801 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.276884 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:48.277126 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.278228 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.279424 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.279973 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280062 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280127 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.280551 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.280590 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260925 14:50:48.280444 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36495 (local address 127.0.0.1:40102)
0925 14:50:48.276332 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.276363 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.276462 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.276906 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.276909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.276910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:48.276982 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.276984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.277962 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.277964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.279166 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.279168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.280179 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.280249 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280252 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.280312 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.280344 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260925 14:50:48.280969 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40935 (local address 127.0.0.1:43316)
0925 14:50:48.276747 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.276755 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.276805 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.277146 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.277147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.277150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:48.277186 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.277187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.278477 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.278478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.279518 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.279520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.280594 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.280661 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280665 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.280853 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.280893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":49,"threads_started":1}
I20260925 14:50:48.281275 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.276747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.1:40935)
0925 14:50:48.276923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.276927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.276936 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:48.276974 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.276975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.276990 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.277061 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:48.277268 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.278336 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.279603 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.280297 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.280359 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.280406 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.281150 (+ 744us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.281197 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260925 14:50:48.281394 4744 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff (DEFAULT_TABLE table=test-table [id=babfbc4e873a4c0595f020428fc248ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:48.281401 4877 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff (DEFAULT_TABLE table=test-table [id=babfbc4e873a4c0595f020428fc248ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:48.281764 4877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee0c30ed91d48f1960412436e6a9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:48.281767 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee0c30ed91d48f1960412436e6a9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:48.282056 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff (DEFAULT_TABLE table=test-table [id=babfbc4e873a4c0595f020428fc248ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:48.282363 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee0c30ed91d48f1960412436e6a9aff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:48.285074 4958 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Bootstrap starting.
I20260925 14:50:48.285429 4959 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Bootstrap starting.
I20260925 14:50:48.285732 4958 tablet_bootstrap.cc:654] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:48.285703 4957 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Bootstrap starting.
I20260925 14:50:48.286093 4959 tablet_bootstrap.cc:654] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:48.286177 4958 log.cc:824] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:48.286592 4957 tablet_bootstrap.cc:654] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:48.286729 4959 log.cc:824] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:48.286855 4958 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: No bootstrap required, opened a new log
I20260925 14:50:48.286911 4958 ts_tablet_manager.cc:1397] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:50:48.287375 4957 log.cc:824] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:48.287524 4959 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: No bootstrap required, opened a new log
I20260925 14:50:48.287587 4959 ts_tablet_manager.cc:1397] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:50:48.288285 4957 tablet_bootstrap.cc:492] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: No bootstrap required, opened a new log
I20260925 14:50:48.288395 4957 ts_tablet_manager.cc:1397] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:50:48.288727 4958 raft_consensus.cc:348] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.288865 4958 raft_consensus.cc:374] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:48.288895 4958 raft_consensus.cc:729] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f58c48cdf14064b8b0f9fc49f406e1, State: Initialized, Role: FOLLOWER
I20260925 14:50:48.289036 4958 consensus_queue.cc:260] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.289302 4810 heartbeater.cc:503] Master 127.0.98.62:45029 was elected leader, sending a full tablet report...
I20260925 14:50:48.289527 4958 ts_tablet_manager.cc:1428] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:50:48.289871 4959 raft_consensus.cc:348] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.290036 4959 raft_consensus.cc:374] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:48.290079 4959 raft_consensus.cc:729] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb5698ff1ee415ca0dd8b8971564eb1, State: Initialized, Role: FOLLOWER
I20260925 14:50:48.290295 4959 consensus_queue.cc:260] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.290627 4677 heartbeater.cc:503] Master 127.0.98.62:45029 was elected leader, sending a full tablet report...
I20260925 14:50:48.290613 4959 ts_tablet_manager.cc:1428] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:48.290937 4957 raft_consensus.cc:348] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.291078 4957 raft_consensus.cc:374] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:48.291122 4957 raft_consensus.cc:729] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d2eddaa43945f2b2e307fe70329710, State: Initialized, Role: FOLLOWER
I20260925 14:50:48.291226 4957 consensus_queue.cc:260] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.291489 4957 ts_tablet_manager.cc:1428] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:50:48.291599 4943 heartbeater.cc:503] Master 127.0.98.62:45029 was elected leader, sending a full tablet report...
W20260925 14:50:48.336953 4811 tablet.cc:2369] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:48.438275 4678 tablet.cc:2369] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:48.502136 4944 tablet.cc:2369] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:48.504959 4964 raft_consensus.cc:482] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:48.505088 4964 raft_consensus.cc:504] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.505475 4964 leader_election.cc:288] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d2eddaa43945f2b2e307fe70329710 (127.0.98.3:36495), 13f58c48cdf14064b8b0f9fc49f406e1 (127.0.98.2:42469)
I20260925 14:50:48.508944 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.505717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36495 (local address 127.0.98.1:57921)
0925 14:50:48.505911 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.505924 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.506026 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.506387 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.506388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.506391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:48.506496 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.506499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.507135 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.507136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.508117 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.508119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.508668 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.508740 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.508744 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.508816 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.508859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":5,"thread_start_us":125,"threads_started":1}
I20260925 14:50:48.509066 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.506045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46217 (local address 127.0.98.2:42469)
0925 14:50:48.506092 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.506094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.506132 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:50:48.506262 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.506263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.506275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.506373 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:48.506564 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.507050 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508161 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.508670 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508775 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.508785 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.508946 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.509000 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:50:48.509087 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.505975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42469 (local address 127.0.98.1:46217)
0925 14:50:48.506070 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.506080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.506140 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.506383 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.506384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.506386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:48.506484 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.506486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.507141 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.507143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.508055 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:48.508057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.508780 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:48.508865 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.508869 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.508941 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.508997 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":69,"thread_start_us":46,"threads_started":1}
I20260925 14:50:48.509079 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.505808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57921 (local address 127.0.98.3:36495)
0925 14:50:48.505890 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.505892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.506004 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:50:48.506175 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.506176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.506187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.506378 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:48.506576 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.507049 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508204 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.508583 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.508668 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.508679 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.508837 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.509000 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:50:48.509371 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee0c30ed91d48f1960412436e6a9aff" candidate_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d2eddaa43945f2b2e307fe70329710" is_pre_election: true
I20260925 14:50:48.509531 4897 raft_consensus.cc:2454] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb5698ff1ee415ca0dd8b8971564eb1 in term 0.
I20260925 14:50:48.509626 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee0c30ed91d48f1960412436e6a9aff" candidate_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" is_pre_election: true
I20260925 14:50:48.509801 4764 raft_consensus.cc:2454] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbb5698ff1ee415ca0dd8b8971564eb1 in term 0.
I20260925 14:50:48.509819 4565 leader_election.cc:302] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d2eddaa43945f2b2e307fe70329710, bbb5698ff1ee415ca0dd8b8971564eb1; no voters:
I20260925 14:50:48.509945 4964 raft_consensus.cc:2781] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:48.509999 4964 raft_consensus.cc:482] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:48.510021 4964 raft_consensus.cc:3037] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:48.536352 4964 raft_consensus.cc:504] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.536574 4964 leader_election.cc:288] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [CANDIDATE]: Term 1 election: Requested vote from peers 14d2eddaa43945f2b2e307fe70329710 (127.0.98.3:36495), 13f58c48cdf14064b8b0f9fc49f406e1 (127.0.98.2:42469)
I20260925 14:50:48.536849 4897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee0c30ed91d48f1960412436e6a9aff" candidate_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d2eddaa43945f2b2e307fe70329710"
I20260925 14:50:48.536931 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee0c30ed91d48f1960412436e6a9aff" candidate_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f58c48cdf14064b8b0f9fc49f406e1"
I20260925 14:50:48.536973 4897 raft_consensus.cc:3037] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:48.537034 4764 raft_consensus.cc:3037] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:48.546269 4897 raft_consensus.cc:2454] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb5698ff1ee415ca0dd8b8971564eb1 in term 1.
I20260925 14:50:48.546269 4764 raft_consensus.cc:2454] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbb5698ff1ee415ca0dd8b8971564eb1 in term 1.
I20260925 14:50:48.546588 4565 leader_election.cc:302] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f58c48cdf14064b8b0f9fc49f406e1, bbb5698ff1ee415ca0dd8b8971564eb1; no voters:
I20260925 14:50:48.546784 4964 raft_consensus.cc:2781] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:48.546993 4964 raft_consensus.cc:686] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [term 1 LEADER]: Becoming Leader. State: Replica: bbb5698ff1ee415ca0dd8b8971564eb1, State: Running, Role: LEADER
I20260925 14:50:48.547118 4964 consensus_queue.cc:237] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } }
I20260925 14:50:48.548121 4487 catalog_manager.cc:5495] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbb5698ff1ee415ca0dd8b8971564eb1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb5698ff1ee415ca0dd8b8971564eb1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40935 } health_report { overall_health: HEALTHY } } }
I20260925 14:50:48.561199 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.559056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43332 (local address 127.0.98.1:40935)
0925 14:50:48.559113 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.559115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.559122 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:50:48.559151 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.559151 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.559168 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.559225 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:48.559337 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.559850 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.560538 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:48.560716 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:48.560790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:48.560895 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:48.560998 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:48.561054 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.561086 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.561102 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:50:48.563052 4897 raft_consensus.cc:1261] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 1 FOLLOWER]: Refusing update from remote peer bbb5698ff1ee415ca0dd8b8971564eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:48.563052 4764 raft_consensus.cc:1261] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 1 FOLLOWER]: Refusing update from remote peer bbb5698ff1ee415ca0dd8b8971564eb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:48.563439 4968 consensus_queue.cc:1035] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14d2eddaa43945f2b2e307fe70329710" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:48.563578 4964 consensus_queue.cc:1035] T 7ee0c30ed91d48f1960412436e6a9aff P bbb5698ff1ee415ca0dd8b8971564eb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f58c48cdf14064b8b0f9fc49f406e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:48.565948 4972 mvcc.cc:205] Tried to move back new op lower bound from 7333264787711553536 to 7333264787649765376. Current Snapshot: MvccSnapshot[applied={T|T < 7333264787711553536}]
I20260925 14:50:48.568517 4487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:48.568596 4487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:48.569370 4487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e19cf948f7124187bc6748967588c4bf: Sending DeleteTablet for 3 replicas of tablet 7ee0c30ed91d48f1960412436e6a9aff
I20260925 14:50:48.569672 4877 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40102
I20260925 14:50:48.569718 4744 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34234
I20260925 14:50:48.569758 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee0c30ed91d48f1960412436e6a9aff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43316
I20260925 14:50:48.569937 4981 tablet_replica.cc:330] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: stopping tablet replica
I20260925 14:50:48.570061 4981 raft_consensus.cc:2229] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:48.570160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4549
I20260925 14:50:48.570176 4981 raft_consensus.cc:2258] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:48.571583 4981 ts_tablet_manager.cc:1905] T 7ee0c30ed91d48f1960412436e6a9aff P 13f58c48cdf14064b8b0f9fc49f406e1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:50:48.575321 4472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40935 (error 108)
I20260925 14:50:48.575626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4682
I20260925 14:50:48.575624 4980 tablet_replica.cc:330] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: stopping tablet replica
I20260925 14:50:48.575757 4980 raft_consensus.cc:2229] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:48.575860 4980 raft_consensus.cc:2258] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:48.576385 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.575689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40935 (local address 127.0.0.1:43348)
0925 14:50:48.576303 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.576344 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592}
W20260925 14:50:48.576530 4472 catalog_manager.cc:4584] TS bbb5698ff1ee415ca0dd8b8971564eb1 (127.0.98.1:40935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ee0c30ed91d48f1960412436e6a9aff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40935: connect: Connection refused (error 111)
I20260925 14:50:48.577126 4980 ts_tablet_manager.cc:1905] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:48.627828 4980 ts_tablet_manager.cc:1918] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:48.627948 4980 log.cc:1196] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/7ee0c30ed91d48f1960412436e6a9aff
I20260925 14:50:48.628230 4980 ts_tablet_manager.cc:1939] T 7ee0c30ed91d48f1960412436e6a9aff P 14d2eddaa43945f2b2e307fe70329710: Deleting consensus metadata
I20260925 14:50:48.629212 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.629070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40935 (local address 127.0.0.1:43350)
0925 14:50:48.629141 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.629179 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:50:48.629758 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.629659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40935 (local address 127.0.0.1:43354)
0925 14:50:48.629707 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.629734 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:50:48.631163 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.631047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42469 (local address 127.0.0.1:34250)
0925 14:50:48.631096 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.631137 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:50:48.631191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4815
I20260925 14:50:48.642042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4456
I20260925 14:50:48.642068 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.641929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36495 (local address 127.0.0.1:40114)
0925 14:50:48.641997 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.642028 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:36495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
2026-09-25T14:50:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2154 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:48Z Disabled control of system clock
I20260925 14:50:48.668864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45963
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:50:48.748440 4987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:48.748637 4987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:48.748658 4987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:48.749732 4987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:48.749768 4987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:48.749780 4987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:48.749792 4987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:48.749804 4987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:48.751142 4987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:48.751343 4987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:48.751567 4987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:50:48.756400 4987 server_base.cc:1034] running on GCE node
W20260925 14:50:48.756379 4995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.756366 4992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.756378 4993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.756839 4987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:48.757107 4987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:48.758260 4987 hybrid_clock.cc:648] HybridClock initialized: now 1790347848758242 us; error 50 us; skew 500 ppm
I20260925 14:50:48.759439 4987 webserver.cc:460] Webserver started at http://127.0.98.62:38811/ using document root <none> and password file <none>
I20260925 14:50:48.759673 4987 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:48.759718 4987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:48.759873 4987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:48.760898 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "0536c2d84f36475db79be92a25f8354d"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "3f009802d0ad48dab4abd69e1884d102"
server_key_iv: "29fad112bf10fea48fd023c9a77e61ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.761260 4987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "0536c2d84f36475db79be92a25f8354d"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "3f009802d0ad48dab4abd69e1884d102"
server_key_iv: "29fad112bf10fea48fd023c9a77e61ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.762629 4987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:48.763360 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.763525 4987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:48.763621 4987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0536c2d84f36475db79be92a25f8354d"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "3f009802d0ad48dab4abd69e1884d102"
server_key_iv: "29fad112bf10fea48fd023c9a77e61ea"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.763698 4987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:48.766871 4987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:48.767124 4987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:48.767266 4987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:48.770862 4987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45963
I20260925 14:50:48.770893 5053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45963 every 8 connection(s)
I20260925 14:50:48.771216 4987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:48.771870 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:48.772795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 4987
I20260925 14:50:48.772905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:48.773069 392 external_mini_cluster.cc:1428] Setting key 152ab228fa8762f09e81fcb432aefb28
I20260925 14:50:48.775386 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: Bootstrap starting.
I20260925 14:50:48.775625 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.773571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.0.98.62:45963)
0925 14:50:48.773757 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.773766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.773798 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:50:48.774327 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.774334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.774407 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.774560 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:48.775225 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:48.775229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:48.775230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:48.775234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:48.775278 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:48.775290 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:48.775371 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.775405 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.775459 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260925 14:50:48.776034 5054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:48.776319 5054 log.cc:824] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:48.776962 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: No bootstrap required, opened a new log
I20260925 14:50:48.778221 5054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260925 14:50:48.778402 5054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:48.778471 5054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0536c2d84f36475db79be92a25f8354d, State: Initialized, Role: FOLLOWER
I20260925 14:50:48.778635 5054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260925 14:50:48.778740 5054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:48.778784 5054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:48.778831 5054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:48.779646 5054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260925 14:50:48.779759 5054 leader_election.cc:302] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0536c2d84f36475db79be92a25f8354d; no voters:
I20260925 14:50:48.779935 5054 leader_election.cc:288] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:48.779982 5059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:48.780130 5059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [term 1 LEADER]: Becoming Leader. State: Replica: 0536c2d84f36475db79be92a25f8354d, State: Running, Role: LEADER
I20260925 14:50:48.780192 5054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:48.780231 5059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } }
I20260925 14:50:48.780609 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0536c2d84f36475db79be92a25f8354d. Latest consensus state: current_term: 1 leader_uuid: "0536c2d84f36475db79be92a25f8354d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } } }
I20260925 14:50:48.780771 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:48.780625 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0536c2d84f36475db79be92a25f8354d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0536c2d84f36475db79be92a25f8354d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45963 } } }
I20260925 14:50:48.780901 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:48.781137 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:48.781612 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:48.783079 5068 catalog_manager.cc:1347] Generated new cluster ID: 34730dcb8ec749c8822eb9e1b822cf73
I20260925 14:50:48.783146 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:48.792426 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:48.792598 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:48.796388 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: Generated new TSK 0
I20260925 14:50:48.796555 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:50:48.801681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:50:48.905588 5078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:48.905788 5078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:48.905810 5078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:48.907146 5078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:48.907186 5078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:48.907212 5078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:48.908564 5078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:48.908804 5078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:48.909127 5078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:48.914770 5084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.914855 5083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:48.914773 5086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:48.915308 5078 server_base.cc:1034] running on GCE node
I20260925 14:50:48.915529 5078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:48.915766 5078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:48.916924 5078 hybrid_clock.cc:648] HybridClock initialized: now 1790347848916902 us; error 47 us; skew 500 ppm
I20260925 14:50:48.918188 5078 webserver.cc:460] Webserver started at http://127.0.98.1:37615/ using document root <none> and password file <none>
I20260925 14:50:48.918486 5078 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:48.918556 5078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:48.918692 5078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:48.919718 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "b152b824f9863fe3425132dee6e3cdd1"
server_key_iv: "d9a0d323a1786f50f6e4cb6b629030b4"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.920084 5078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "b152b824f9863fe3425132dee6e3cdd1"
server_key_iv: "d9a0d323a1786f50f6e4cb6b629030b4"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.921396 5078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:48.922134 5092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.922304 5078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:48.922410 5078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2"
format_stamp: "Formatted at 2026-09-25 14:50:48 on dist-test-slave-gpc1"
server_key: "b152b824f9863fe3425132dee6e3cdd1"
server_key_iv: "d9a0d323a1786f50f6e4cb6b629030b4"
server_key_version: "encryptionkey@0"
I20260925 14:50:48.922479 5078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:48.925717 5078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:48.926005 5078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:48.926142 5078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:48.926395 5078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:48.926771 5078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:48.926807 5078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.926827 5078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:48.926839 5078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:48.933851 5078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33517
I20260925 14:50:48.933889 5205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33517 every 8 connection(s)
I20260925 14:50:48.934202 5078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:48.936826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5078
I20260925 14:50:48.936928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:48.937093 392 external_mini_cluster.cc:1428] Setting key 9b78920ed3ac15c9687b18f4ccc9e7fb
I20260925 14:50:48.937287 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.934892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57147 (local address 127.0.98.62:45963)
0925 14:50:48.934951 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:48.934953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:48.935215 (+ 262us) server_negotiation.cc:407] Connection header received
0925 14:50:48.935682 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:48.935683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:48.935689 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:48.935776 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:48.936989 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:48.936991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:48.936991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:48.936996 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:48.937012 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:48.937016 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:48.937045 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:48.937202 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:50:48.937219 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:50:48.937543 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:48.934878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.1:57147)
0925 14:50:48.935087 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:48.935127 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:48.935583 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:48.935826 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:48.935829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:48.935836 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:48.936796 (+ 960us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:48.936797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:48.936823 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:48.936851 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:48.937091 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:48.937095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:48.937098 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:48.937210 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:50:48.937270 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":120,"threads_started":1}
I20260925 14:50:48.938088 5206 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260925 14:50:48.938175 5206 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:48.938174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:50:48.938333 5206 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
I20260925 14:50:48.938845 5018 ts_manager.cc:194] Registered new tserver with Master: dbd4db8d9a6d4cd8b523cd91acf308f2 (127.0.98.1:33517)
W20260925 14:50:49.018008 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:49.018266 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:49.018303 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:49.019477 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:49.019544 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:49.019584 5209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:49.020903 5209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:49.021160 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:49.021449 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:49.026443 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.026419 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.026477 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.026702 5209 server_base.cc:1034] running on GCE node
I20260925 14:50:49.026906 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:49.027122 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:49.028285 5209 hybrid_clock.cc:648] HybridClock initialized: now 1790347849028272 us; error 29 us; skew 500 ppm
I20260925 14:50:49.029523 5209 webserver.cc:460] Webserver started at http://127.0.98.2:34655/ using document root <none> and password file <none>
I20260925 14:50:49.029768 5209 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:49.029825 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:49.029978 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:49.031121 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "94df72c5c2194e1da310f9c391b892b9"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7165acc23de3460375e320f7f2541965"
server_key_iv: "554563536141255509e295756a657565"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.031517 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "94df72c5c2194e1da310f9c391b892b9"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7165acc23de3460375e320f7f2541965"
server_key_iv: "554563536141255509e295756a657565"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.032840 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:50:49.033525 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.033697 5209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:49.033790 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "94df72c5c2194e1da310f9c391b892b9"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7165acc23de3460375e320f7f2541965"
server_key_iv: "554563536141255509e295756a657565"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.033859 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:49.037151 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:49.037400 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:49.037536 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:49.037735 5209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:49.038023 5209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:49.038054 5209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.038101 5209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:49.038129 5209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.043978 5209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41589
I20260925 14:50:49.044021 5336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41589 every 8 connection(s)
I20260925 14:50:49.044317 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:49.047154 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.045036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57803 (local address 127.0.98.62:45963)
0925 14:50:49.045094 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.045097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.045363 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:50:49.045790 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.045791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.045797 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.045865 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.046955 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.046956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.046956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.046960 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.046973 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.046976 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.047025 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.047091 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.047103 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:49.047400 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.045031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.2:57803)
0925 14:50:49.045208 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.045240 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.045711 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.045917 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.045919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.045927 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.046819 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.046820 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.046846 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.046874 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.047038 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.047042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.047044 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.047100 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.047146 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260925 14:50:49.047851 5337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260925 14:50:49.047943 5337 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:49.048077 5337 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
I20260925 14:50:49.048465 5018 ts_manager.cc:194] Registered new tserver with Master: 94df72c5c2194e1da310f9c391b892b9 (127.0.98.2:41589)
I20260925 14:50:49.053084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5209
I20260925 14:50:49.053217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:49.053455 392 external_mini_cluster.cc:1428] Setting key 5b4f86e817c96c295fc90addd87e334f
I20260925 14:50:49.054615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:50:49.134634 5340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:49.134855 5340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:49.134888 5340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:49.136045 5340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:49.136106 5340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:49.136152 5340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:49.137581 5340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:49.137804 5340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:49.138082 5340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:49.143075 5346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.143103 5348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.143081 5345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.143313 5340 server_base.cc:1034] running on GCE node
I20260925 14:50:49.143553 5340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:49.143740 5340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:49.144873 5340 hybrid_clock.cc:648] HybridClock initialized: now 1790347849144847 us; error 41 us; skew 500 ppm
I20260925 14:50:49.146034 5340 webserver.cc:460] Webserver started at http://127.0.98.3:41359/ using document root <none> and password file <none>
I20260925 14:50:49.146256 5340 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:49.146299 5340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:49.146464 5340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:49.147468 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "be1b337df5e6467c97d70f66614b7c5d"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7371d09df5ec271663d960f1c15f1cb1"
server_key_iv: "46eacf1f0c65047d22d3289a2ca2cccb"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.147814 5340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be1b337df5e6467c97d70f66614b7c5d"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7371d09df5ec271663d960f1c15f1cb1"
server_key_iv: "46eacf1f0c65047d22d3289a2ca2cccb"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.149046 5340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:50:49.149703 5354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.149885 5340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:49.149974 5340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "be1b337df5e6467c97d70f66614b7c5d"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "7371d09df5ec271663d960f1c15f1cb1"
server_key_iv: "46eacf1f0c65047d22d3289a2ca2cccb"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.150043 5340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:49.153290 5340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:49.153533 5340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:49.153663 5340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:49.153880 5340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:49.154174 5340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:49.154204 5340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.154258 5340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:49.154287 5340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.159788 5340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38545
I20260925 14:50:49.159880 5467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38545 every 8 connection(s)
I20260925 14:50:49.160193 5340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:49.162921 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.160887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35543 (local address 127.0.98.62:45963)
0925 14:50:49.160973 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.160975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.161170 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:50:49.161602 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.161604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.161609 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.161660 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.162746 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.162747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.162748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.162752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.162765 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.162768 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.162814 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.162859 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.162870 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:50:49.163146 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.160856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45963 (local address 127.0.98.3:35543)
0925 14:50:49.161055 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.161086 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.161515 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.161711 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.161714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.161721 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.162619 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.162620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.162646 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.162672 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.162812 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.162814 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.162817 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.162866 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.162908 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":111,"threads_started":1}
I20260925 14:50:49.163658 5468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45963
I20260925 14:50:49.163727 5468 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:49.163879 5468 heartbeater.cc:511] Master 127.0.98.62:45963 requested a full tablet report, sending...
I20260925 14:50:49.164260 5018 ts_manager.cc:194] Registered new tserver with Master: be1b337df5e6467c97d70f66614b7c5d (127.0.98.3:38545)
I20260925 14:50:49.168758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5340
I20260925 14:50:49.168850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:49.168999 392 external_mini_cluster.cc:1428] Setting key 595bfab7dfc60d3c49f34adbeb75369b
I20260925 14:50:49.169762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:49.172166 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.171037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.0.98.62:45963)
0925 14:50:49.171096 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.171098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.171203 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:50:49.171279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.171280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.171284 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.171328 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.171968 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.171969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.171970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.171974 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.171987 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.171990 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.172039 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.172087 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.172101 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:50:49.173717 5018 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:49.175318 5018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:49.179186 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58482 (local address 127.0.98.2:41589)
0925 14:50:49.177822 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.177826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.177848 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:50:49.177874 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.177876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.177885 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.177955 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.178915 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.178919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.178919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.178922 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.178947 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.178960 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.179026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.179115 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.179127 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":80,"threads_started":1}
I20260925 14:50:49.179199 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41589 (local address 127.0.0.1:58482)
0925 14:50:49.177546 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.177584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.177640 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.177975 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.177976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.177977 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.178575 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.178576 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.178590 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.178611 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.179028 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.179035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.179038 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.179104 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.179117 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260925 14:50:49.179199 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33517 (local address 127.0.0.1:41092)
0925 14:50:49.178211 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.178221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.178263 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.178472 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.178473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.178475 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.178825 (+ 350us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.178826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.178830 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.178837 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.179081 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.179082 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.179083 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.179124 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.179133 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":20,"thread_start_us":90,"threads_started":1}
I20260925 14:50:49.179495 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.0.98.1:33517)
0925 14:50:49.177792 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.177797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.178322 (+ 525us) server_negotiation.cc:407] Connection header received
0925 14:50:49.178372 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.178374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.178384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.178459 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.178911 (+ 452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.178918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.178918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.178922 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.178957 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.178987 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.179058 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.179228 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.179411 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260925 14:50:49.179682 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38545 (local address 127.0.0.1:53366)
0925 14:50:49.177709 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.177719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.177757 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.177923 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.177947 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.177953 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.178701 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.178703 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.178729 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.178738 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.179570 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.179570 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.179571 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.179617 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.179629 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":91,"threads_started":1}
I20260925 14:50:49.179740 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.177593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.3:38545)
0925 14:50:49.177739 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.177743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.177756 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:50:49.177823 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.177826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.177836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.177919 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.178910 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.178913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.178913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.178917 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.178946 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.178960 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.179026 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.179643 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.179657 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260925 14:50:49.179924 5271 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f (DEFAULT_TABLE table=test-table [id=a6d615307e7b4ef3a1f86eeadc48e7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:49.180074 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f (DEFAULT_TABLE table=test-table [id=a6d615307e7b4ef3a1f86eeadc48e7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:49.180325 5271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa98486e90e4d64a3e4a960b3a8c34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.180404 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa98486e90e4d64a3e4a960b3a8c34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.183651 5483 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Bootstrap starting.
I20260925 14:50:49.184103 5402 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f (DEFAULT_TABLE table=test-table [id=a6d615307e7b4ef3a1f86eeadc48e7b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:49.184211 5483 tablet_bootstrap.cc:654] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.184383 5402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa98486e90e4d64a3e4a960b3a8c34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.184525 5483 log.cc:824] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.184828 5482 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: Bootstrap starting.
I20260925 14:50:49.185237 5483 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: No bootstrap required, opened a new log
I20260925 14:50:49.185302 5483 ts_tablet_manager.cc:1397] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:49.185566 5482 tablet_bootstrap.cc:654] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.185843 5482 log.cc:824] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.186523 5482 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: No bootstrap required, opened a new log
I20260925 14:50:49.186596 5482 ts_tablet_manager.cc:1397] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:49.187667 5486 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Bootstrap starting.
I20260925 14:50:49.187824 5483 raft_consensus.cc:348] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.187996 5483 raft_consensus.cc:374] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.188036 5483 raft_consensus.cc:729] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94df72c5c2194e1da310f9c391b892b9, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.188134 5483 consensus_queue.cc:260] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.188326 5486 tablet_bootstrap.cc:654] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.188364 5483 ts_tablet_manager.cc:1428] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:49.188495 5337 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
I20260925 14:50:49.188782 5486 log.cc:824] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.188805 5482 raft_consensus.cc:348] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.188973 5482 raft_consensus.cc:374] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.189039 5482 raft_consensus.cc:729] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd4db8d9a6d4cd8b523cd91acf308f2, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.189168 5482 consensus_queue.cc:260] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.189414 5482 ts_tablet_manager.cc:1428] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:50:49.189462 5206 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
I20260925 14:50:49.189577 5486 tablet_bootstrap.cc:492] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: No bootstrap required, opened a new log
I20260925 14:50:49.189640 5486 ts_tablet_manager.cc:1397] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:49.191578 5486 raft_consensus.cc:348] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.191694 5486 raft_consensus.cc:374] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.191715 5486 raft_consensus.cc:729] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1b337df5e6467c97d70f66614b7c5d, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.191820 5486 consensus_queue.cc:260] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.192029 5486 ts_tablet_manager.cc:1428] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:50:49.192150 5468 heartbeater.cc:503] Master 127.0.98.62:45963 was elected leader, sending a full tablet report...
W20260925 14:50:49.294873 5338 tablet.cc:2369] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:49.339277 5490 raft_consensus.cc:482] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:49.339407 5490 raft_consensus.cc:504] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.339821 5490 leader_election.cc:288] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94df72c5c2194e1da310f9c391b892b9 (127.0.98.2:41589), dbd4db8d9a6d4cd8b523cd91acf308f2 (127.0.98.1:33517)
I20260925 14:50:49.341205 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.340056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57999 (local address 127.0.98.2:41589)
0925 14:50:49.340102 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.340104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.340133 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:50:49.340226 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.340228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.340234 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.340350 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.340938 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.340939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.340939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.340945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.340963 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.340969 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.341065 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.341121 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.341135 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:49.341212 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.339993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41589 (local address 127.0.98.3:57999)
0925 14:50:49.340056 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.340068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.340106 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.340353 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.340354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.340356 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.340810 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.340810 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.340816 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.340828 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.341067 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.341068 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.341069 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.341120 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.341133 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:50:49.341456 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.340105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38759 (local address 127.0.98.1:33517)
0925 14:50:49.340173 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.340175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.340298 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:50:49.340326 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.340328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.340334 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.340392 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.341174 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.341175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.341175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.341181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.341196 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.341200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.341260 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.341359 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.341377 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:50:49.341616 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.340024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33517 (local address 127.0.98.3:38759)
0925 14:50:49.340219 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.340227 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.340277 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.340410 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.340411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.340424 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:49.341035 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:50:49.341036 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:49.341042 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:50:49.341052 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:49.341269 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:49.341269 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:49.341270 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:50:49.341532 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 14:50:49.341544 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":127,"threads_started":1}
I20260925 14:50:49.341552 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa98486e90e4d64a3e4a960b3a8c34f" candidate_uuid: "be1b337df5e6467c97d70f66614b7c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94df72c5c2194e1da310f9c391b892b9" is_pre_election: true
I20260925 14:50:49.341722 5291 raft_consensus.cc:2454] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1b337df5e6467c97d70f66614b7c5d in term 0.
I20260925 14:50:49.341952 5356 leader_election.cc:302] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94df72c5c2194e1da310f9c391b892b9, be1b337df5e6467c97d70f66614b7c5d; no voters:
I20260925 14:50:49.341938 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa98486e90e4d64a3e4a960b3a8c34f" candidate_uuid: "be1b337df5e6467c97d70f66614b7c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" is_pre_election: true
I20260925 14:50:49.342051 5160 raft_consensus.cc:2454] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be1b337df5e6467c97d70f66614b7c5d in term 0.
I20260925 14:50:49.342101 5490 raft_consensus.cc:2781] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:49.342156 5490 raft_consensus.cc:482] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:49.342192 5490 raft_consensus.cc:3037] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.342905 5490 raft_consensus.cc:504] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.343043 5490 leader_election.cc:288] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [CANDIDATE]: Term 1 election: Requested vote from peers 94df72c5c2194e1da310f9c391b892b9 (127.0.98.2:41589), dbd4db8d9a6d4cd8b523cd91acf308f2 (127.0.98.1:33517)
I20260925 14:50:49.343180 5160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa98486e90e4d64a3e4a960b3a8c34f" candidate_uuid: "be1b337df5e6467c97d70f66614b7c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2"
I20260925 14:50:49.343240 5160 raft_consensus.cc:3037] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.343942 5160 raft_consensus.cc:2454] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1b337df5e6467c97d70f66614b7c5d in term 1.
I20260925 14:50:49.344098 5358 leader_election.cc:302] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be1b337df5e6467c97d70f66614b7c5d, dbd4db8d9a6d4cd8b523cd91acf308f2; no voters:
I20260925 14:50:49.344081 5291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa98486e90e4d64a3e4a960b3a8c34f" candidate_uuid: "be1b337df5e6467c97d70f66614b7c5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94df72c5c2194e1da310f9c391b892b9"
I20260925 14:50:49.344170 5291 raft_consensus.cc:3037] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.344198 5490 raft_consensus.cc:2781] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:49.344362 5490 raft_consensus.cc:686] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 1 LEADER]: Becoming Leader. State: Replica: be1b337df5e6467c97d70f66614b7c5d, State: Running, Role: LEADER
I20260925 14:50:49.344458 5490 consensus_queue.cc:237] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } }
I20260925 14:50:49.345081 5291 raft_consensus.cc:2454] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be1b337df5e6467c97d70f66614b7c5d in term 1.
I20260925 14:50:49.345356 5018 catalog_manager.cc:5495] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d reported cstate change: term changed from 0 to 1, leader changed from <none> to be1b337df5e6467c97d70f66614b7c5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "be1b337df5e6467c97d70f66614b7c5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1b337df5e6467c97d70f66614b7c5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:49.359179 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.358220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.98.3:38545)
0925 14:50:49.358333 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.358335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.358384 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:50:49.358415 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.358416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.358441 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.358494 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.358993 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.358994 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.358995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.358998 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.359011 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.359014 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.359063 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.359110 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.359120 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:50:49.360965 5291 raft_consensus.cc:1261] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 1 FOLLOWER]: Refusing update from remote peer be1b337df5e6467c97d70f66614b7c5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:49.361001 5160 raft_consensus.cc:1261] T 6aa98486e90e4d64a3e4a960b3a8c34f P dbd4db8d9a6d4cd8b523cd91acf308f2 [term 1 FOLLOWER]: Refusing update from remote peer be1b337df5e6467c97d70f66614b7c5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:49.361343 5490 consensus_queue.cc:1035] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd4db8d9a6d4cd8b523cd91acf308f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:49.361495 5492 consensus_queue.cc:1035] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "94df72c5c2194e1da310f9c391b892b9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:49.363783 5495 mvcc.cc:205] Tried to move back new op lower bound from 7333264790979801088 to 7333264790915551232. Current Snapshot: MvccSnapshot[applied={T|T < 7333264790979801088}]
I20260925 14:50:49.363837 5497 mvcc.cc:205] Tried to move back new op lower bound from 7333264790979801088 to 7333264790915551232. Current Snapshot: MvccSnapshot[applied={T|T < 7333264790979801088}]
I20260925 14:50:49.365859 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:49.365950 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:49.366744 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0536c2d84f36475db79be92a25f8354d: Sending DeleteTablet for 3 replicas of tablet 6aa98486e90e4d64a3e4a960b3a8c34f
I20260925 14:50:49.367044 5271 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:49 UTC) from {username='slave'} at 127.0.0.1:58482
I20260925 14:50:49.367111 5140 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:49 UTC) from {username='slave'} at 127.0.0.1:41092
I20260925 14:50:49.367357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5078
I20260925 14:50:49.367375 5505 tablet_replica.cc:330] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: stopping tablet replica
I20260925 14:50:49.367547 5505 raft_consensus.cc:2229] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:49.367687 5505 raft_consensus.cc:2258] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:49.368175 5505 ts_tablet_manager.cc:1905] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:49.370246 5505 ts_tablet_manager.cc:1918] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:49.370362 5505 log.cc:1196] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/6aa98486e90e4d64a3e4a960b3a8c34f
I20260925 14:50:49.370620 5505 ts_tablet_manager.cc:1939] T 6aa98486e90e4d64a3e4a960b3a8c34f P 94df72c5c2194e1da310f9c391b892b9: Deleting consensus metadata
I20260925 14:50:49.371033 5003 catalog_manager.cc:4841] TS 94df72c5c2194e1da310f9c391b892b9 (127.0.98.2:41589): tablet 6aa98486e90e4d64a3e4a960b3a8c34f (table test-table [id=a6d615307e7b4ef3a1f86eeadc48e7b0]) successfully deleted
I20260925 14:50:49.371377 5402 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa98486e90e4d64a3e4a960b3a8c34f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:49 UTC) from {username='slave'} at 127.0.0.1:53366
I20260925 14:50:49.371592 5507 tablet_replica.cc:330] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: stopping tablet replica
I20260925 14:50:49.371703 5507 raft_consensus.cc:2229] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:49.371794 5507 raft_consensus.cc:2258] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:49.372151 5507 ts_tablet_manager.cc:1905] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:49.373502 5507 ts_tablet_manager.cc:1918] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:49.373586 5507 log.cc:1196] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/6aa98486e90e4d64a3e4a960b3a8c34f
I20260925 14:50:49.373827 5507 ts_tablet_manager.cc:1939] T 6aa98486e90e4d64a3e4a960b3a8c34f P be1b337df5e6467c97d70f66614b7c5d: Deleting consensus metadata
I20260925 14:50:49.374210 5004 catalog_manager.cc:4841] TS be1b337df5e6467c97d70f66614b7c5d (127.0.98.3:38545): tablet 6aa98486e90e4d64a3e4a960b3a8c34f (table test-table [id=a6d615307e7b4ef3a1f86eeadc48e7b0]) successfully deleted
W20260925 14:50:49.375168 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33517 (error 108)
I20260925 14:50:49.375672 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.375522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33517 (local address 127.0.0.1:41106)
0925 14:50:49.375575 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.375614 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:50:49.375727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5209
W20260925 14:50:49.378944 5005 catalog_manager.cc:4584] TS dbd4db8d9a6d4cd8b523cd91acf308f2 (127.0.98.1:33517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6aa98486e90e4d64a3e4a960b3a8c34f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33517: connect: Connection refused (error 111)
I20260925 14:50:49.381803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5340
I20260925 14:50:49.385651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 4987
2026-09-25T14:50:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (742 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:50:49.405082 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:50:49.406103 392 webserver.cc:460] Webserver started at https://127.0.0.1:40881/ using document root <none> and password file <none>
I20260925 14:50:49.406220 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:50:49.406463 392 webserver.cc:460] Webserver started at http://127.0.0.1:45849/ using document root <none> and password file <none>
2026-09-25T14:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:49Z Disabled control of system clock
I20260925 14:50:49.413738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43843
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43843
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40881/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/testchainca.cert with env {}
W20260925 14:50:49.491799 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:49.492004 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:49.492026 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:49.493111 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:49.493144 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:49.493156 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:49.493170 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:49.493181 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:50:49.493193 5514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:50:49.493204 5514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40881/jwks.json
I20260925 14:50:49.494663 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40881/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:49.494860 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:49.495018 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:49.499744 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.499779 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.499850 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.499853 5514 server_base.cc:1034] running on GCE node
I20260925 14:50:49.500250 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:49.500561 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:49.501727 5514 hybrid_clock.cc:648] HybridClock initialized: now 1790347849501711 us; error 47 us; skew 500 ppm
I20260925 14:50:49.502956 5514 webserver.cc:460] Webserver started at http://127.0.98.62:44935/ using document root <none> and password file <none>
I20260925 14:50:49.503203 5514 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:49.503273 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:49.503436 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:49.504504 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "c67258df30c54744b5c8d8da1a36c147"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "ad420464bf38ed2262eebef2549f4fb4"
server_key_iv: "c96aa5a29047d4be3437ef177bf42325"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.504859 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "c67258df30c54744b5c8d8da1a36c147"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "ad420464bf38ed2262eebef2549f4fb4"
server_key_iv: "c96aa5a29047d4be3437ef177bf42325"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.506167 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:49.507014 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.507213 5514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:49.507336 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "c67258df30c54744b5c8d8da1a36c147"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "ad420464bf38ed2262eebef2549f4fb4"
server_key_iv: "c96aa5a29047d4be3437ef177bf42325"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.507431 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:49.516039 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:49.516341 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:49.516484 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:49.520154 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43843
I20260925 14:50:49.520217 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43843 every 8 connection(s)
I20260925 14:50:49.520553 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:49.521500 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.523736 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: Bootstrap starting.
I20260925 14:50:49.524267 5581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.524499 5581 log.cc:824] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.525126 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: No bootstrap required, opened a new log
I20260925 14:50:49.526676 5581 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 14:50:49.526811 5581 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.526844 5581 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67258df30c54744b5c8d8da1a36c147, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.526976 5581 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 14:50:49.527052 5581 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:49.527076 5581 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:49.527102 5581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.527592 5581 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 14:50:49.527693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5514
I20260925 14:50:49.527709 5581 leader_election.cc:302] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67258df30c54744b5c8d8da1a36c147; no voters:
I20260925 14:50:49.527783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:49.527889 5581 leader_election.cc:288] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:49.527956 5584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:49.528021 392 external_mini_cluster.cc:1428] Setting key 87682e4e9512c70848c494d87eb5659e
I20260925 14:50:49.528146 5584 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [term 1 LEADER]: Becoming Leader. State: Replica: c67258df30c54744b5c8d8da1a36c147, State: Running, Role: LEADER
I20260925 14:50:49.528170 5581 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:49.528264 5584 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } }
I20260925 14:50:49.528581 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67258df30c54744b5c8d8da1a36c147. Latest consensus state: current_term: 1 leader_uuid: "c67258df30c54744b5c8d8da1a36c147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } } }
I20260925 14:50:49.528653 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:49.528851 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67258df30c54744b5c8d8da1a36c147" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67258df30c54744b5c8d8da1a36c147" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43843 } } }
I20260925 14:50:49.528949 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147 [sys.catalog]: This master's current role is: LEADER
W20260925 14:50:49.530283 5602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:49.530366 5602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:49.530419 5593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:49.530579 5593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:49.533033 5593 catalog_manager.cc:1347] Generated new cluster ID: 00a236f39e2b45b6aafb5194d0132a04
I20260925 14:50:49.533088 5593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:49.534777 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.528629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.0.98.62:43843)
0925 14:50:49.529307 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.529317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.529394 (+ 77us) server_negotiation.cc:407] Connection header received
0925 14:50:49.529525 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.529533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.529616 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.529802 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.530013 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.531500 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.532652 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.533215 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.533339 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.534065 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.534069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.534069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.534267 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.534308 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.534322 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.534413 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.534503 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.534577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":110,"threads_started":1}
I20260925 14:50:49.535617 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:50:49.551482 5593 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:49.552054 5593 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:49.562160 5593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67258df30c54744b5c8d8da1a36c147: Generated new TSK 0
I20260925 14:50:49.562388 5593 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:50:49.583804 5611 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:50:49.583909 5611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:50:49.584142 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.568647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39618 (local address 127.0.98.62:43843)
0925 14:50:49.568705 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.568707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.568835 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:50:49.568911 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.568912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.568925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.569018 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:50:49.569153 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.569909 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.570719 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.571025 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.571098 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.571305 (+ 207us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:50:49.583931 (+ 12626us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:50:49.584070 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 14:50:49.584199 5592 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39618: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:50:49.584637 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:49.568565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43843 (local address 127.0.0.1:39618)
0925 14:50:49.568769 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.568779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.568835 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.569035 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.569036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.569037 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:50:49.569090 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:49.569092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.569989 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.569989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:49.570644 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:49.570646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.571113 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.571121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:49.571175 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.571193 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:50:49.584107 (+ 12914us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:50:49.584338 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40881/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:50:49.585088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5514
2026-09-25T14:50:49Z chronyd exiting
I20260925 14:50:49.620810 392 test_util.cc:175] -----------------------------------------------
I20260925 14:50:49.620903 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347825060285-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:50:49.621695 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:50:49.622412 392 webserver.cc:460] Webserver started at https://127.0.0.1:40261/ using document root <none> and password file <none>
I20260925 14:50:49.622473 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:50:49.622635 392 webserver.cc:460] Webserver started at http://127.0.0.1:40715/ using document root <none> and password file <none>
2026-09-25T14:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:49Z Disabled control of system clock
I20260925 14:50:49.630002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40699
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40261/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/testchainca.cert with env {}
W20260925 14:50:49.708083 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:49.708290 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:49.708312 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:49.709836 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:49.709882 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:49.709895 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:49.709908 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:49.709973 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:50:49.709993 5617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:50:49.710016 5617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40261/jwks.json
I20260925 14:50:49.711664 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40261/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:49.711930 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:49.712162 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:50:49.717017 5617 server_base.cc:1034] running on GCE node
W20260925 14:50:49.716986 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.716961 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.716971 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.717502 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:49.717787 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:49.718923 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790347849718912 us; error 32 us; skew 500 ppm
I20260925 14:50:49.720131 5617 webserver.cc:460] Webserver started at http://127.0.98.62:41741/ using document root <none> and password file <none>
I20260925 14:50:49.720376 5617 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:49.720445 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:49.720592 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:49.721606 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "545a30fd1c2341079bd17eec11ce91ee"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "fb36fd3463c9ba2c31fda6c4808876fd"
server_key_iv: "898b85a7543c2002d806f79809a348d8"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.721964 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "545a30fd1c2341079bd17eec11ce91ee"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "fb36fd3463c9ba2c31fda6c4808876fd"
server_key_iv: "898b85a7543c2002d806f79809a348d8"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.723415 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:49.724141 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.724299 5617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:49.724414 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "545a30fd1c2341079bd17eec11ce91ee"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "fb36fd3463c9ba2c31fda6c4808876fd"
server_key_iv: "898b85a7543c2002d806f79809a348d8"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.724486 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:49.750676 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:49.751016 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:49.751178 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:49.754982 5617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40699
I20260925 14:50:49.755023 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40699 every 8 connection(s)
I20260925 14:50:49.755394 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:49.756359 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.758569 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee: Bootstrap starting.
I20260925 14:50:49.759198 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.759482 5684 log.cc:824] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.760136 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee: No bootstrap required, opened a new log
I20260925 14:50:49.761677 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260925 14:50:49.761812 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.761835 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545a30fd1c2341079bd17eec11ce91ee, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.762006 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260925 14:50:49.762070 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:49.762107 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:49.762156 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.762743 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260925 14:50:49.762856 5684 leader_election.cc:302] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 545a30fd1c2341079bd17eec11ce91ee; no voters:
I20260925 14:50:49.763033 5684 leader_election.cc:288] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:49.763056 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:49.763275 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [term 1 LEADER]: Becoming Leader. State: Replica: 545a30fd1c2341079bd17eec11ce91ee, State: Running, Role: LEADER
I20260925 14:50:49.763278 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:49.763381 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } }
I20260925 14:50:49.763739 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "545a30fd1c2341079bd17eec11ce91ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } } }
I20260925 14:50:49.763833 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:49.763851 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 545a30fd1c2341079bd17eec11ce91ee. Latest consensus state: current_term: 1 leader_uuid: "545a30fd1c2341079bd17eec11ce91ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "545a30fd1c2341079bd17eec11ce91ee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40699 } } }
I20260925 14:50:49.763911 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:49.764055 5695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:49.764178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5617
I20260925 14:50:49.764283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:49.764488 392 external_mini_cluster.cc:1428] Setting key d11cd71e49e390061bd78ceeaaa25cd7
I20260925 14:50:49.764593 5695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:49.766582 5695 catalog_manager.cc:1347] Generated new cluster ID: b89ae380934d44c9b8343cded8f69899
I20260925 14:50:49.766635 5695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:49.770303 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.764906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.98.62:40699)
0925 14:50:49.765530 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.765539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.766097 (+ 558us) server_negotiation.cc:407] Connection header received
0925 14:50:49.766249 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.766254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.766308 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.766481 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.766702 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.767581 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.768390 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.768801 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.768943 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.769643 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.769646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.769647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.769820 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.769859 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.769871 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.769942 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.770016 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.770091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":67,"threads_started":1}
I20260925 14:50:49.771178 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:50:49.789731 5695 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:49.790273 5695 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:49.795919 5695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 545a30fd1c2341079bd17eec11ce91ee: Generated new TSK 0
I20260925 14:50:49.796110 5695 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:50:49.831193 5714 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:50:49.831321 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:50:49.831473 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.809406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.0.98.62:40699)
0925 14:50:49.809453 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.809455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.809564 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:50:49.809634 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.809635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.809645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.809704 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:50:49.809844 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.810381 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.811551 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.811738 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.811783 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.811969 (+ 186us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:50:49.831228 (+ 19259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:50:49.831398 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:50:49.831537 5700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34626: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 14:50:49.831859 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:49.809321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40699 (local address 127.0.0.1:34626)
0925 14:50:49.809489 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:49.809501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:49.809568 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:49.809720 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:49.809721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:49.809723 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:50:49.809781 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:49.809784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.810487 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.810488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:49.811466 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:49.811469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.811801 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.811808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:49.811876 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.811881 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:50:49.831643 (+ 19762us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:50:49.831760 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40261/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260925 14:50:49.832805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5617
2026-09-25T14:50:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:50:49.870075 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:50:49.870872 392 webserver.cc:460] Webserver started at https://127.0.0.1:44071/ using document root <none> and password file <none>
I20260925 14:50:49.870936 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:50:49.871102 392 webserver.cc:460] Webserver started at http://127.0.0.1:44277/ using document root <none> and password file <none>
2026-09-25T14:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:49Z Disabled control of system clock
I20260925 14:50:49.879122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37851
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37851
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:50:49.957729 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:49.957921 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:49.957942 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:49.959029 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:49.959064 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:49.959077 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:49.959089 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:49.959101 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:50:49.959113 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:50:49.959124 5720 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44071/jwks.json
I20260925 14:50:49.960386 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:49.960575 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:49.960795 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:49.965473 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:49.965487 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.965526 5720 server_base.cc:1034] running on GCE node
W20260925 14:50:49.965440 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:49.965816 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:49.966149 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:49.967308 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790347849967303 us; error 37 us; skew 500 ppm
I20260925 14:50:49.968564 5720 webserver.cc:460] Webserver started at http://127.0.98.62:35973/ using document root <none> and password file <none>
I20260925 14:50:49.968807 5720 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:49.968889 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:49.969051 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:49.970055 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "7351347d151f4756a5e451db436eb6b5"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "70c4c53870d6fe660d0d3377af8a13c5"
server_key_iv: "7fb68f398b09798ebaa7b3d72d685379"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.970466 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "7351347d151f4756a5e451db436eb6b5"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "70c4c53870d6fe660d0d3377af8a13c5"
server_key_iv: "7fb68f398b09798ebaa7b3d72d685379"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.971776 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:49.972462 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:49.972644 5720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:49.972744 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "7351347d151f4756a5e451db436eb6b5"
format_stamp: "Formatted at 2026-09-25 14:50:49 on dist-test-slave-gpc1"
server_key: "70c4c53870d6fe660d0d3377af8a13c5"
server_key_iv: "7fb68f398b09798ebaa7b3d72d685379"
server_key_version: "encryptionkey@0"
I20260925 14:50:49.972836 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:49.981261 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:49.981530 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:49.981683 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:49.985427 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37851
I20260925 14:50:49.985491 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37851 every 8 connection(s)
I20260925 14:50:49.985800 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:49.986536 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:49.989003 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5: Bootstrap starting.
I20260925 14:50:49.989563 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:49.989845 5787 log.cc:824] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:49.990549 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5: No bootstrap required, opened a new log
I20260925 14:50:49.992066 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } }
I20260925 14:50:49.992198 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:49.992220 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7351347d151f4756a5e451db436eb6b5, State: Initialized, Role: FOLLOWER
I20260925 14:50:49.992413 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } }
I20260925 14:50:49.992477 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:49.992498 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:49.992524 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:49.992985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5720
I20260925 14:50:49.993086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:49.993211 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } }
I20260925 14:50:49.993278 392 external_mini_cluster.cc:1428] Setting key 5aeeef125afcd44c2727195d85a039ef
I20260925 14:50:49.993322 5787 leader_election.cc:302] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7351347d151f4756a5e451db436eb6b5; no voters:
I20260925 14:50:49.993489 5787 leader_election.cc:288] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:49.993546 5790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:49.993741 5790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [term 1 LEADER]: Becoming Leader. State: Replica: 7351347d151f4756a5e451db436eb6b5, State: Running, Role: LEADER
I20260925 14:50:49.993755 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:49.993875 5790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } }
I20260925 14:50:49.994226 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7351347d151f4756a5e451db436eb6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } } }
I20260925 14:50:49.994321 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:49.994390 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7351347d151f4756a5e451db436eb6b5. Latest consensus state: current_term: 1 leader_uuid: "7351347d151f4756a5e451db436eb6b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7351347d151f4756a5e451db436eb6b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37851 } } }
I20260925 14:50:49.994468 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:49.995149 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:49.995884 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:49.997486 5800 catalog_manager.cc:1347] Generated new cluster ID: 2433508acbb6460ba94e72dbddc6ff88
I20260925 14:50:49.997545 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:49.999819 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:49.993807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.0.98.62:37851)
0925 14:50:49.994609 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:49.994617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:49.995106 (+ 489us) server_negotiation.cc:407] Connection header received
0925 14:50:49.995272 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:49.995280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:49.995362 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:49.995577 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:49.995793 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.996910 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.997846 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:49.998269 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:49.998439 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:49.999127 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:49.999131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:49.999131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:49.999312 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:49.999345 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:49.999368 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:49.999443 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:49.999524 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:50:49.999607 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":61,"threads_started":1}
I20260925 14:50:50.000730 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:50:50.005015 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:50.005625 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:50.015653 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7351347d151f4756a5e451db436eb6b5: Generated new TSK 0
I20260925 14:50:50.015857 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:50:50.026537 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:50.022956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.0.98.62:37851)
0925 14:50:50.023024 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:50.023026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:50.023037 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:50:50.023071 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:50.023072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:50.023086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:50.023176 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:50:50.023280 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.024004 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.024811 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.025116 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.025197 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:50.025318 (+ 121us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:50:50.026292 (+ 974us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:50:50.026457 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260925 14:50:50.026602 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49862: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:50:50.028441 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:50.022674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37851 (local address 127.0.0.1:49862)
0925 14:50:50.022824 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:50.022834 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:50.022913 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:50.023179 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:50.023181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:50.023182 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:50:50.023229 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:50.023231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.024117 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.024119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:50.024720 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:50.024722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.025194 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.025200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:50.025239 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:50.025243 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:50:50.026414 (+ 1171us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:50:50.028344 (+ 1930us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37851: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1734,"thread_start_us":105,"threads_started":1}
I20260925 14:50:50.028749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5720
E20260925 14:50:50.037568 5817 webserver.cc:533] Webserver: sslize failed:
E20260925 14:50:50.037631 5817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T14:50:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (195 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:50Z Disabled control of system clock
I20260925 14:50:50.072829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39661
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39661
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/testchainca.cert with env {}
W20260925 14:50:50.151046 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:50.151270 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:50.151309 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:50.152487 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:50.152554 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:50.152575 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:50.152594 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:50.152626 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:50:50.152647 5821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:50:50.152668 5821 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:50:50.154105 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:50.154393 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:50.154603 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:50:50.159363 5821 server_base.cc:1034] running on GCE node
W20260925 14:50:50.159334 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:50.159334 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:50.159320 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:50.159766 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:50.160092 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:50.161252 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790347850161231 us; error 34 us; skew 500 ppm
I20260925 14:50:50.162531 5821 webserver.cc:460] Webserver started at http://127.0.98.62:36389/ using document root <none> and password file <none>
I20260925 14:50:50.162760 5821 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:50.162827 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:50.162961 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:50.163972 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf"
format_stamp: "Formatted at 2026-09-25 14:50:50 on dist-test-slave-gpc1"
server_key: "c13af440ddac9e4bce68c1dbdc3c100e"
server_key_iv: "3aa907ebcc1d67e014cc41643e42cc81"
server_key_version: "encryptionkey@0"
I20260925 14:50:50.164341 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf"
format_stamp: "Formatted at 2026-09-25 14:50:50 on dist-test-slave-gpc1"
server_key: "c13af440ddac9e4bce68c1dbdc3c100e"
server_key_iv: "3aa907ebcc1d67e014cc41643e42cc81"
server_key_version: "encryptionkey@0"
I20260925 14:50:50.165664 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:50.166412 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:50.166569 5821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:50.166656 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf"
format_stamp: "Formatted at 2026-09-25 14:50:50 on dist-test-slave-gpc1"
server_key: "c13af440ddac9e4bce68c1dbdc3c100e"
server_key_iv: "3aa907ebcc1d67e014cc41643e42cc81"
server_key_version: "encryptionkey@0"
I20260925 14:50:50.166709 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:50.182716 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:50.182956 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:50.183054 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:50.186836 5821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39661
I20260925 14:50:50.186893 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39661 every 8 connection(s)
I20260925 14:50:50.187214 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:50.188105 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:50.190308 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf: Bootstrap starting.
I20260925 14:50:50.190888 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:50.191112 5888 log.cc:824] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:50.191701 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf: No bootstrap required, opened a new log
I20260925 14:50:50.193234 5888 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } }
I20260925 14:50:50.193375 5888 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:50.193397 5888 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca885e64e3a42cbb2aab34aa6e0a4cf, State: Initialized, Role: FOLLOWER
I20260925 14:50:50.193531 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } }
I20260925 14:50:50.193612 5888 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:50.193634 5888 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:50.193660 5888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:50.194125 5888 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } }
I20260925 14:50:50.194204 5888 leader_election.cc:302] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca885e64e3a42cbb2aab34aa6e0a4cf; no voters:
I20260925 14:50:50.194334 5888 leader_election.cc:288] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:50.194413 5891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:50.194556 5891 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [term 1 LEADER]: Becoming Leader. State: Replica: eca885e64e3a42cbb2aab34aa6e0a4cf, State: Running, Role: LEADER
I20260925 14:50:50.194660 5888 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:50.194661 5891 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } }
I20260925 14:50:50.195088 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } } }
I20260925 14:50:50.195197 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca885e64e3a42cbb2aab34aa6e0a4cf. Latest consensus state: current_term: 1 leader_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca885e64e3a42cbb2aab34aa6e0a4cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39661 } } }
I20260925 14:50:50.195279 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:50.195190 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:50.195712 5900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:50.196180 5900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:50.196913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5821
I20260925 14:50:50.197027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:50.197206 392 external_mini_cluster.cc:1428] Setting key eb10de6af786b461e442ebf1f6163a24
I20260925 14:50:50.197829 5900 catalog_manager.cc:1347] Generated new cluster ID: 6f1d8f04b2cc47ed9d9d303c8a1b3c28
I20260925 14:50:50.197881 5900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:50.204226 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:50.197648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.0.98.62:39661)
0925 14:50:50.198444 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:50.198453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:50.198487 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:50:50.198597 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:50.198604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:50.198671 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:50.198805 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:50.199120 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.200544 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.201654 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.202256 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.202408 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:50.203286 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:50.203290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:50.203290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:50:50.203516 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:50.203556 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:50:50.203570 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:50.203630 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:50.203826 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 14:50:50.203900 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260925 14:50:50.205103 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:50:50.213124 5900 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:50.213613 5900 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:50.221483 5900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eca885e64e3a42cbb2aab34aa6e0a4cf: Generated new TSK 0
I20260925 14:50:50.221645 5900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:50:50.255152 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:50.229233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.98.62:39661)
0925 14:50:50.229292 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:50.229294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:50.229459 (+ 165us) server_negotiation.cc:407] Connection header received
0925 14:50:50.229525 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:50.229526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:50.229538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:50.229607 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:50:50.229707 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.230521 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.231295 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.231635 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.231706 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:50.231900 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:50:50.254893 (+ 22993us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:50:50.255076 (+ 183us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 14:50:50.255228 5911 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:50:50.255401 5917 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:50.229137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39661 (local address 127.0.0.1:48104)
0925 14:50:50.229380 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:50.229392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:50.229457 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:50.229614 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:50.229615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:50.229616 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:50:50.229660 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:50.229663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.230620 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.230621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:50.231210 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:50.231211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:50.231732 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:50.231736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:50.231796 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:50.231802 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:50:50.255041 (+ 23239us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:50:50.255156 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":193,"threads_started":1}
I20260925 14:50:50.255743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5821
2026-09-25T14:50:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (207 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:50 dist-test-slave-gpc1 krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:50 dist-test-slave-gpc1 krb5kdc[5929](info): set up 2 sockets
Sep 25 14:50:50 dist-test-slave-gpc1 krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260925 14:50:51.470677 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:51 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:51.584699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45191
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:51.663007 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:51.663198 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:51.663219 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:51.664280 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:51.664319 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:51.664335 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:51.664346 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:51.664358 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:51.665737 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:51.665910 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:51.666064 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:51.670856 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:51.670892 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:51.670930 5942 server_base.cc:1034] running on GCE node
W20260925 14:50:51.670832 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:51.671299 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:51.671602 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:51.672786 5942 hybrid_clock.cc:648] HybridClock initialized: now 1790347851672765 us; error 36 us; skew 500 ppm
Sep 25 14:50:51 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:51.675443 5942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:51.675839 5942 webserver.cc:460] Webserver started at http://127.0.98.62:37483/ using document root <none> and password file <none>
I20260925 14:50:51.676019 5942 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:51.676062 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:51.676151 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:51.677186 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "d36129f0261141efbcfe9cedfebb04be"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "4f176c63707ea7b6f31c762e10bf1a80"
server_key_iv: "5edf1d9662000e7c2274d770f1e3345d"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.677510 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "d36129f0261141efbcfe9cedfebb04be"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "4f176c63707ea7b6f31c762e10bf1a80"
server_key_iv: "5edf1d9662000e7c2274d770f1e3345d"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.678894 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:51.679713 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:51.679888 5942 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:51.679998 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "d36129f0261141efbcfe9cedfebb04be"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "4f176c63707ea7b6f31c762e10bf1a80"
server_key_iv: "5edf1d9662000e7c2274d770f1e3345d"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.680078 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:51.696221 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:51.698055 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:51.698235 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:51.702210 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45191
I20260925 14:50:51.702244 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45191 every 8 connection(s)
I20260925 14:50:51.702831 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:51.703464 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:51.705736 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be: Bootstrap starting.
I20260925 14:50:51.706311 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:51.706637 6010 log.cc:824] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:51.707319 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be: No bootstrap required, opened a new log
I20260925 14:50:51.708849 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } }
I20260925 14:50:51.708995 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:51.709019 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d36129f0261141efbcfe9cedfebb04be, State: Initialized, Role: FOLLOWER
I20260925 14:50:51.709172 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } }
I20260925 14:50:51.709244 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:51.709266 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:51.709331 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:51.709522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 5942
I20260925 14:50:51.709641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:51.709844 392 external_mini_cluster.cc:1428] Setting key 653d46495a548d9cd9365c043a9530aa
I20260925 14:50:51.710548 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } }
I20260925 14:50:51.710654 6010 leader_election.cc:302] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d36129f0261141efbcfe9cedfebb04be; no voters:
I20260925 14:50:51.710793 6010 leader_election.cc:288] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:51.710829 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:51.710981 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [term 1 LEADER]: Becoming Leader. State: Replica: d36129f0261141efbcfe9cedfebb04be, State: Running, Role: LEADER
I20260925 14:50:51.711035 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:51.711081 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } }
I20260925 14:50:51.711483 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [sys.catalog]: SysCatalogTable state changed. Reason: New leader d36129f0261141efbcfe9cedfebb04be. Latest consensus state: current_term: 1 leader_uuid: "d36129f0261141efbcfe9cedfebb04be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } } }
I20260925 14:50:51.711566 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:51.711498 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d36129f0261141efbcfe9cedfebb04be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d36129f0261141efbcfe9cedfebb04be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45191 } } }
I20260925 14:50:51.711625 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:51.712150 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:51.712658 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:51.715488 6024 catalog_manager.cc:1347] Generated new cluster ID: 72c1fb1491884471bcda832a3f92db4b
I20260925 14:50:51.715545 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:51 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:51.721369 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:51.710384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.98.62:45191)
0925 14:50:51.710590 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:51.710599 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:51.710633 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:50:51.710747 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:51.710757 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:51.710813 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:51.710995 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:51.711921 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.713394 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.714206 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.714820 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.714931 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:51.717639 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:51.717643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:51.717643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:51.717655 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:51.719771 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:51.720148 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:51.720151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:51.720151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:51.720279 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:51.720483 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:51.720484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:51.720485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:51.720811 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:51.720967 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:51.721038 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:50:51.721172 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260925 14:50:51.728317 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:51.729001 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:51.741863 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d36129f0261141efbcfe9cedfebb04be: Generated new TSK 0
I20260925 14:50:51.742033 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:51.776530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:51.859525 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:51.859719 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:51.859741 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:51.860818 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:51.860850 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:51.860875 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:51.862233 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:51.862455 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:51.862644 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:51.867475 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:51.867511 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:51.867633 6036 server_base.cc:1034] running on GCE node
W20260925 14:50:51.867699 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:51.868018 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:51.868327 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:51.869498 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790347851869492 us; error 36 us; skew 500 ppm
Sep 25 14:50:51 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:51.872356 6036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:51.872716 6036 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260925 14:50:51.872885 6036 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:51.872925 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:51.873008 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:51.873939 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "4e49f46da05346f5b34d192d687c833e"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "ce0b5bcb15ec2ad4568aeda5a3823131"
server_key_iv: "ea33cf27e8f939e5472511923f28036c"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.874224 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4e49f46da05346f5b34d192d687c833e"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "ce0b5bcb15ec2ad4568aeda5a3823131"
server_key_iv: "ea33cf27e8f939e5472511923f28036c"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.875553 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:51.876236 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:51.876416 6036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:51.876513 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4e49f46da05346f5b34d192d687c833e"
format_stamp: "Formatted at 2026-09-25 14:50:51 on dist-test-slave-gpc1"
server_key: "ce0b5bcb15ec2ad4568aeda5a3823131"
server_key_iv: "ea33cf27e8f939e5472511923f28036c"
server_key_version: "encryptionkey@0"
I20260925 14:50:51.876598 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:51.887308 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:51.889870 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:51.890044 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:51.890285 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:51.890720 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:51.890784 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:51.890828 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:51.890887 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:51.897420 6036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42221
I20260925 14:50:51.897497 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42221 every 8 connection(s)
I20260925 14:50:51.897835 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:51.901427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6036
I20260925 14:50:51.901535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:51.901713 392 external_mini_cluster.cc:1428] Setting key e42171e13fc600fe7ca0c78f89a81b1b
Sep 25 14:50:51 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347851, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:51.908329 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:51.898610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51845 (local address 127.0.98.62:45191)
0925 14:50:51.898667 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:51.898669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:51.898889 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:50:51.899865 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:51.899866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:51.899881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:51.899932 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:51.900534 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.901039 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.902461 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.902675 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.902759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:51.905235 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:51.905237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:51.905237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:51.905244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:51.906748 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:51.907265 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:51.907265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:51.907266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:51.907313 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:51.907635 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:51.907636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:51.907636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:51.907765 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:51.907832 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:51.908189 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 14:50:51.908266 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:51.908594 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:51.898580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45191 (local address 127.0.98.1:51845)
0925 14:50:51.898761 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:51.898810 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:51.899739 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:51.900013 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:51.900017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:51.900250 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:51.900435 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:51.900440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.901133 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.901135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:51.902364 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:51.902367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:51.902767 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:51.902768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:51.902895 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:51.903416 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:51.903424 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:51.905138 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:51.906832 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:51.906840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:51.906842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:51.907180 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:51.907384 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:51.907385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:51.907386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:51.907536 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:51.907850 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:51.907856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:51.908030 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 14:50:51.908164 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:50:51.908337 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260925 14:50:51.909207 6165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45191
I20260925 14:50:51.909318 6165 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:51.909504 6165 heartbeater.cc:511] Master 127.0.98.62:45191 requested a full tablet report, sending...
I20260925 14:50:51.910039 5974 ts_manager.cc:194] Registered new tserver with Master: 4e49f46da05346f5b34d192d687c833e (127.0.98.1:42221)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:50:51.910943 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:51.927213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:52.005609 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:52.005844 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:52.005882 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:52.007083 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:52.007145 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:52.007189 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:52.008638 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:52.008908 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:52.009143 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:52.014194 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:52.014285 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:52.014298 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:52.014611 6170 server_base.cc:1034] running on GCE node
I20260925 14:50:52.014856 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:52.015089 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:52.016248 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790347852016247 us; error 40 us; skew 500 ppm
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.018800 6170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:52.019338 6170 webserver.cc:460] Webserver started at http://127.0.98.2:43127/ using document root <none> and password file <none>
I20260925 14:50:52.019587 6170 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:52.019636 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:52.019788 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:52.020915 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "38f96618fe0449d3950b6da2e0c53ddb"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "d38f1d607b930338c98f0afec96e82b6"
server_key_iv: "7f9092aeef0b4946ddfeafebd6b7beb2"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.021297 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "38f96618fe0449d3950b6da2e0c53ddb"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "d38f1d607b930338c98f0afec96e82b6"
server_key_iv: "7f9092aeef0b4946ddfeafebd6b7beb2"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.022650 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:52.023379 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.023548 6170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:52.023648 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "38f96618fe0449d3950b6da2e0c53ddb"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "d38f1d607b930338c98f0afec96e82b6"
server_key_iv: "7f9092aeef0b4946ddfeafebd6b7beb2"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.023727 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:52.032464 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:52.034214 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:52.034435 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:52.034648 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:52.034950 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:52.035004 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.035048 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:52.035079 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.041314 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42037
I20260925 14:50:52.041358 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42037 every 8 connection(s)
I20260925 14:50:52.041682 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:52.050835 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.042427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53449 (local address 127.0.98.62:45191)
0925 14:50:52.042480 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.042484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.042706 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:50:52.043619 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.043621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.043631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.043689 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.044284 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.044767 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.045703 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.045882 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.045933 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.048159 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:52.048160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:52.048160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:52.048168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:52.049519 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.049892 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.049893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.049893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.049952 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.050177 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.050178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.050178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.050312 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:52.050472 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:52.050694 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:50:52.050770 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:52.050987 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.042411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45191 (local address 127.0.98.2:53449)
0925 14:50:52.042581 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:52.042613 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:52.043503 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:52.043759 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:52.043762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:52.044016 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:52.044193 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:52.044197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.044856 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.044857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:52.045632 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:52.045634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.045950 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.045951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:52.046051 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.046605 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:52.046613 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:52.048071 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:52.049596 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:52.049601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:52.049603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:52.049818 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:52.050018 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:52.050019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:52.050019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:52.050111 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:52.050459 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:52.050463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:52.050573 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:50:52.050658 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:50:52.050772 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260925 14:50:52.051297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6170
I20260925 14:50:52.051421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:52.051644 392 external_mini_cluster.cc:1428] Setting key f9a5374a51b92912e3a520d4e344a89c
I20260925 14:50:52.051641 6299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45191
I20260925 14:50:52.051714 6299 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:52.051918 6299 heartbeater.cc:511] Master 127.0.98.62:45191 requested a full tablet report, sending...
I20260925 14:50:52.052345 5974 ts_manager.cc:194] Registered new tserver with Master: 38f96618fe0449d3950b6da2e0c53ddb (127.0.98.2:42037)
I20260925 14:50:52.052691 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53449
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:52.076704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:52.155560 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:52.155784 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:52.155828 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:52.156957 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:52.157019 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:52.157063 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:52.158526 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:52.158778 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:52.158999 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:52.164006 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:52.164089 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:52.164006 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:52.164179 6304 server_base.cc:1034] running on GCE node
I20260925 14:50:52.164388 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:52.164613 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:52.165772 6304 hybrid_clock.cc:648] HybridClock initialized: now 1790347852165754 us; error 47 us; skew 500 ppm
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.168160 6304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:52.168548 6304 webserver.cc:460] Webserver started at http://127.0.98.3:37071/ using document root <none> and password file <none>
I20260925 14:50:52.168773 6304 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:52.168839 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:52.168999 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:52.169992 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c3fc6d2e15b4df98407a0991471017e"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "4420fbf97c8cd595e56a331754a3f1d3"
server_key_iv: "5a58ccd66def364a88a3bc44f2c71c82"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.170394 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c3fc6d2e15b4df98407a0991471017e"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "4420fbf97c8cd595e56a331754a3f1d3"
server_key_iv: "5a58ccd66def364a88a3bc44f2c71c82"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.171689 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:52.172379 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.172536 6304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:52.172629 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "6c3fc6d2e15b4df98407a0991471017e"
format_stamp: "Formatted at 2026-09-25 14:50:52 on dist-test-slave-gpc1"
server_key: "4420fbf97c8cd595e56a331754a3f1d3"
server_key_iv: "5a58ccd66def364a88a3bc44f2c71c82"
server_key_version: "encryptionkey@0"
I20260925 14:50:52.172724 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:52.179647 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:52.181438 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:52.181586 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:52.181808 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:52.182083 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:52.182140 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.182178 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:52.182204 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:52.187860 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36759
I20260925 14:50:52.187927 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36759 every 8 connection(s)
I20260925 14:50:52.188231 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:52.191004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6304
I20260925 14:50:52.191076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:52.191255 392 external_mini_cluster.cc:1428] Setting key 6e0ad1d356a6ffbfcf40193d7e89dbf9
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:52.197597 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.188986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39315 (local address 127.0.98.62:45191)
0925 14:50:52.189081 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.189083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.189267 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:50:52.190205 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.190206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.190227 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.190280 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.190940 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.191630 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.192761 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.192941 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.193009 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.195096 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:52.195097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:52.195098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:52.195104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:52.196355 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.196716 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.196716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.196717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.196762 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.196999 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.197000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.197000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.197111 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:52.197175 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:52.197474 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 14:50:52.197538 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:50:52.197621 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.188948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45191 (local address 127.0.98.3:39315)
0925 14:50:52.189142 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:52.189185 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:52.190094 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:52.190361 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:52.190365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:52.190623 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:52.190829 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:52.190834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.191724 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.191725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:52.192690 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:52.192692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.193025 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.193026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:52.193140 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.193657 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:52.193664 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:52.195009 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:52.196431 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:52.196437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:52.196439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:52.196654 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:52.196826 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:52.196827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:52.196827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:52.196916 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:52.197174 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:52.197178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:52.197281 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 14:50:52.197361 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:50:52.197473 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260925 14:50:52.198107 6433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45191
I20260925 14:50:52.198207 6433 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:52.198338 6433 heartbeater.cc:511] Master 127.0.98.62:45191 requested a full tablet report, sending...
I20260925 14:50:52.198733 5974 ts_manager.cc:194] Registered new tserver with Master: 6c3fc6d2e15b4df98407a0991471017e (127.0.98.3:36759)
I20260925 14:50:52.199100 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39315
I20260925 14:50:52.203203 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/test-admin.keytab.
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.218945 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:50:52.220518 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:52.219799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42644 (local address 127.0.98.1:42221)
0925 14:50:52.219985 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.219989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.219999 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:52.220024 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.220026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.220039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.220101 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.220429 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42644: BlockingRecv error: recv got EOF from 127.0.0.1:42644 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
W20260925 14:50:52.220566 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.221547 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.222488 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:52.229856 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.222986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.98.1:42221)
0925 14:50:52.223061 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.223063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.223100 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:50:52.223153 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.223153 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.223163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.223209 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.223587 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.224287 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.224941 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.225225 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.225274 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.227181 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:52.227188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:52.227188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:52.227194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:52.228841 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.229146 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.229148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.229149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.229199 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.229352 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.229353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.229353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.229473 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:52.229568 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:52.229628 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:50:52.229686 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260925 14:50:52.231643 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:52.231160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.1:42221)
0925 14:50:52.231203 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.231204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.231215 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:50:52.231262 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.231262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.231272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.231329 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.231583 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42666: BlockingRecv error: recv got EOF from 127.0.0.1:42666 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:50:52.231734 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc)
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.232764 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.233668 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:52.240964 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.234206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.98.1:42221)
0925 14:50:52.234292 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.234294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.234306 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:50:52.234515 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.234516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.234524 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.234574 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.234925 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.235386 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.236032 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.236215 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.236265 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.238561 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:52.238562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:52.238562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:52.238568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:52.239911 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.240238 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.240239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.240239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.240284 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.240451 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.240452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.240452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.240534 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:52.240602 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:52.240757 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:50:52.240821 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 14:50:52.242583 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:50:52.242017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.98.1:42221)
0925 14:50:52.242053 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.242055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.242088 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:50:52.242136 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.242137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.242144 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.242185 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.242542 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42688: BlockingRecv error: recv got EOF from 127.0.0.1:42688 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:50:52.242657 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347825060285-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.243525 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:52.244361 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:50:52.251132 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:52.244897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.0.98.1:42221)
0925 14:50:52.244934 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:52.244935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:52.244999 (+ 64us) server_negotiation.cc:407] Connection header received
0925 14:50:52.245031 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:52.245032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:52.245039 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:52.245081 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:52.245412 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.245858 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.246591 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:52.246773 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:52.246842 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:52.248898 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:52.248899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:52.248899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:52.248905 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:52.250266 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.250604 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.250605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.250605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.250652 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:52.250802 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:52.250802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:52.250803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:52.250879 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:52.250942 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:52.251022 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:50:52.251072 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:50:52.251868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6036
I20260925 14:50:52.256765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6170
I20260925 14:50:52.260723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6304
I20260925 14:50:52.264953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 5942
2026-09-25T14:50:52Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1839 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[6453](info): set up 2 sockets
Sep 25 14:50:52 dist-test-slave-gpc1 krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260925 14:50:53.323011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:53 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 14:50:53.439627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:53.522333 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:53.522634 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:53.522684 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:53.523874 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:53.523954 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:53.523983 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:53.524015 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:53.524048 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:53.525547 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:53.525852 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:53.526099 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:53.531194 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:53.531265 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:53.531452 6466 server_base.cc:1034] running on GCE node
W20260925 14:50:53.531219 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:53.531713 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:53.531951 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:53.533092 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790347853533075 us; error 31 us; skew 500 ppm
Sep 25 14:50:53 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:53.539902 6466 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 14:50:53.540417 6466 webserver.cc:460] Webserver started at http://127.0.98.62:44391/ using document root <none> and password file <none>
I20260925 14:50:53.540663 6466 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:53.540742 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:53.540905 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:53.541990 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "517c35e223d647748517bc8379274670"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "33f022306e2773625227954d1ddf2c6b"
server_key_iv: "7dd37d6c5a5ba84fd45656430a902bde"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.542456 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "517c35e223d647748517bc8379274670"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "33f022306e2773625227954d1ddf2c6b"
server_key_iv: "7dd37d6c5a5ba84fd45656430a902bde"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.543866 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:53.544675 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:53.544855 6466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:53.544961 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "517c35e223d647748517bc8379274670"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "33f022306e2773625227954d1ddf2c6b"
server_key_iv: "7dd37d6c5a5ba84fd45656430a902bde"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.545053 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:53.558245 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:53.560406 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:53.560585 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:53.564988 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45659
I20260925 14:50:53.565032 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45659 every 8 connection(s)
I20260925 14:50:53.565384 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:53.566270 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:53.568542 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Bootstrap starting.
I20260925 14:50:53.569114 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:53.569410 6534 log.cc:824] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:53.570075 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: No bootstrap required, opened a new log
I20260925 14:50:53.571648 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:50:53.571782 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:53.571805 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 517c35e223d647748517bc8379274670, State: Initialized, Role: FOLLOWER
I20260925 14:50:53.571969 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:50:53.572044 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:53.572088 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:53.572140 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:53.572924 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:50:53.573031 6534 leader_election.cc:302] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 517c35e223d647748517bc8379274670; no voters:
I20260925 14:50:53.573196 6534 leader_election.cc:288] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:53.573230 6537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:53.573417 6537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 LEADER]: Becoming Leader. State: Replica: 517c35e223d647748517bc8379274670, State: Running, Role: LEADER
I20260925 14:50:53.573470 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:53.573515 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:50:53.573909 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } }
I20260925 14:50:53.573906 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517c35e223d647748517bc8379274670. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } }
I20260925 14:50:53.574378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6466
I20260925 14:50:53.574483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:53.574127 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:53.574702 392 external_mini_cluster.cc:1428] Setting key 19da081a440d5948780dbf6737f50641
I20260925 14:50:53.574110 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
W20260925 14:50:53.575307 6553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:53.575546 6553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:53.575618 6552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:53.575822 6552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:53.577448 6552 catalog_manager.cc:1347] Generated new cluster ID: 30232186eaf64c3a811ff2153377b04d
I20260925 14:50:53.577505 6552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:50:53 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:53.584051 6552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:53.584721 6552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:53.585172 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:53.575165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.0.98.62:45659)
0925 14:50:53.575432 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:53.575441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:53.575478 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:50:53.575699 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:53.575707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:53.575763 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:53.575929 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:53.576677 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.577910 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.578735 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.579148 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.579266 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:53.581526 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:53.581529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:53.581530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:53.581541 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:53.583273 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:53.583628 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:53.583631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:53.583631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:53.583741 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:53.583997 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:53.583998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:53.583998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:53.584274 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:53.584773 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:53.584842 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:50:53.584974 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260925 14:50:53.586867 6552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Generated new TSK 0
I20260925 14:50:53.587031 6552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 14:50:53.619637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:53.712785 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:53.712981 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:53.713004 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:53.714210 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:53.714255 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:53.714282 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:53.715840 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:53.716071 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:53.716307 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:53.721222 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:53.721307 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:53.721232 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:53.721482 6560 server_base.cc:1034] running on GCE node
I20260925 14:50:53.721805 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:53.722077 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:53.723256 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790347853723236 us; error 32 us; skew 500 ppm
Sep 25 14:50:53 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:53.725919 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:50:53.726470 6560 webserver.cc:460] Webserver started at http://127.0.98.1:42673/ using document root <none> and password file <none>
I20260925 14:50:53.726703 6560 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:53.726770 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:53.726904 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:53.727955 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c862c5525d4478496446ad8ecf53d52"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "f4268524572f298dfc88a3ab4ec792fd"
server_key_iv: "652bf6233a201daea0f426245e48df39"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.728317 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c862c5525d4478496446ad8ecf53d52"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "f4268524572f298dfc88a3ab4ec792fd"
server_key_iv: "652bf6233a201daea0f426245e48df39"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.729679 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:53.730513 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:53.730707 6560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:53.730808 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4c862c5525d4478496446ad8ecf53d52"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "f4268524572f298dfc88a3ab4ec792fd"
server_key_iv: "652bf6233a201daea0f426245e48df39"
server_key_version: "encryptionkey@0"
I20260925 14:50:53.730895 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:53.750617 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:53.753044 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:53.753206 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:53.753471 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:53.753784 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:53.753839 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:53.753882 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:53.753914 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:53.759763 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32957
I20260925 14:50:53.759867 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32957 every 8 connection(s)
I20260925 14:50:53.760151 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:53.764923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6560
I20260925 14:50:53.765039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:53.765259 392 external_mini_cluster.cc:1428] Setting key de0caf0e7d0503a7d6a2898164edb8d7
Sep 25 14:50:53 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:53.770056 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:53.760918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41813 (local address 127.0.98.62:45659)
0925 14:50:53.760989 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:53.760991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:53.761170 (+ 179us) server_negotiation.cc:407] Connection header received
0925 14:50:53.761661 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:53.761662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:53.761672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:53.761719 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:53.762655 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.763144 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.764110 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.764314 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.764371 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:53.767016 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:53.767017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:53.767017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:53.767024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:53.768648 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:53.769026 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:53.769027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:53.769027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:53.769089 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:53.769310 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:53.769310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:53.769311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:53.769461 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:53.769536 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:53.769837 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:50:53.769935 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:50:53.770079 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:53.760904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:41813)
0925 14:50:53.761065 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:53.761101 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:53.761579 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:53.761784 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:53.761787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:53.762379 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:53.762578 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:53.762583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.763222 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.763223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:53.764030 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:53.764032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:53.764389 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:53.764390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:53.764483 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:53.765074 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:53.765084 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:53.766932 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:53.768722 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:53.768727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:53.768729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:53.768953 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:53.769154 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:53.769154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:53.769154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:53.769245 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:53.769528 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:53.769532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:53.769642 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:50:53.769790 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 14:50:53.769921 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260925 14:50:53.770601 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:53.770717 6689 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:53.770891 6689 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:53.771425 6498 ts_manager.cc:194] Registered new tserver with Master: 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
I20260925 14:50:53.772069 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41813
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
W20260925 14:50:53.953445 392 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.0.98.2: real 0.187s user 0.000s sys 0.003s
I20260925 14:50:53.953819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:54.032833 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:54.033021 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:54.033043 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:54.034168 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:54.034207 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:54.034235 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:54.035702 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:54.036000 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:54.036304 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.041237 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.041379 6694 server_base.cc:1034] running on GCE node
W20260925 14:50:54.041215 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.041240 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.041633 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.041889 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.043047 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790347854043035 us; error 28 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.045576 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:50:54.046000 6694 webserver.cc:460] Webserver started at http://127.0.98.2:43645/ using document root <none> and password file <none>
I20260925 14:50:54.046213 6694 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.046283 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.046447 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:54.047459 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "13d9857b70164976ae22a6faf92b58cc"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "9c38c14b6079e97375534255a0f588f6"
server_key_iv: "90c70973dc3d33c59c876477122f5339"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.047873 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "13d9857b70164976ae22a6faf92b58cc"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "9c38c14b6079e97375534255a0f588f6"
server_key_iv: "90c70973dc3d33c59c876477122f5339"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.049288 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:54.050041 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.050294 6694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:54.050443 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "13d9857b70164976ae22a6faf92b58cc"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "9c38c14b6079e97375534255a0f588f6"
server_key_iv: "90c70973dc3d33c59c876477122f5339"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.050529 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.078210 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.080811 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:54.080988 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.081248 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:54.081590 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:54.081647 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.081686 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:54.081736 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.088397 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37901
I20260925 14:50:54.088449 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37901 every 8 connection(s)
I20260925 14:50:54.088806 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:54.097628 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.089539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41559 (local address 127.0.98.62:45659)
0925 14:50:54.089576 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.089578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.089787 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:50:54.090287 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.090288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.090297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.090432 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.091366 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.091840 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.092862 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.093030 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.093073 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.095248 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.095250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.095250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.095257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.096508 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.096872 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.096873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.096873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.096919 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.097122 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.097122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.097123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.097213 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.097260 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.097501 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.097565 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:50:54.097826 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.089512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:41559)
0925 14:50:54.089670 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.089713 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.090204 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.090499 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.090502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.091072 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.091275 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.091280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.091922 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.091923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.092773 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.092776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.093097 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.093098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.093227 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.093762 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.093770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.095132 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.096589 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.096594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.096595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.096809 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.096976 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.096976 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.096976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.097070 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.097286 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.097289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.097395 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.097475 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.097587 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 14:50:54.098421 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:54.098532 6823 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:54.098704 6823 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:54.099020 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6694
I20260925 14:50:54.099092 6498 ts_manager.cc:194] Registered new tserver with Master: 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901)
I20260925 14:50:54.099141 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:54.099341 392 external_mini_cluster.cc:1428] Setting key b612eb614a53c3595f79687f8adfa2dc
I20260925 14:50:54.099520 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41559
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:50:54.124794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:54.204375 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:54.204581 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:54.204603 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:54.205721 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:54.205761 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:54.205811 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:54.207217 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:54.207479 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:54.207697 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.212630 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.212734 6828 server_base.cc:1034] running on GCE node
W20260925 14:50:54.212661 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.212626 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.213070 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.213305 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.214447 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790347854214435 us; error 39 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.216802 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:50:54.217235 6828 webserver.cc:460] Webserver started at http://127.0.98.3:42753/ using document root <none> and password file <none>
I20260925 14:50:54.217415 6828 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.217454 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.217537 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:54.218559 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "394d329381b2487283d737bbcc863193"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "3296411fa1ae286832393697a8535bad"
server_key_iv: "18272fcfd699c98ee7a57d02d438187b"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.218930 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "394d329381b2487283d737bbcc863193"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "3296411fa1ae286832393697a8535bad"
server_key_iv: "18272fcfd699c98ee7a57d02d438187b"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.220150 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:50:54.220762 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.220917 6828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:54.221000 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "394d329381b2487283d737bbcc863193"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "3296411fa1ae286832393697a8535bad"
server_key_iv: "18272fcfd699c98ee7a57d02d438187b"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.221065 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.233623 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.235962 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:54.236116 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.236330 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:54.236629 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:54.236667 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.236688 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:54.236733 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.242240 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35201
I20260925 14:50:54.242293 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35201 every 8 connection(s)
I20260925 14:50:54.242652 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:54.249436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6828
I20260925 14:50:54.249539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:54.249758 392 external_mini_cluster.cc:1428] Setting key 18bc6b358b84024218131cbd82797187
I20260925 14:50:54.251637 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.243316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38015 (local address 127.0.98.62:45659)
0925 14:50:54.243362 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.243365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.243584 (+ 219us) server_negotiation.cc:407] Connection header received
0925 14:50:54.244030 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.244031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.244041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.244093 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.244927 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.245399 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.246329 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.246559 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.246613 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.248764 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.248765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.248765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.248771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.250442 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.250885 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.250886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.250886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.250936 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.251140 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.251140 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.251141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.251234 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.251281 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.251515 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.251575 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:54.251830 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.243325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:38015)
0925 14:50:54.243471 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.243504 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.243946 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.244142 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.244145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.244679 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.244856 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.244860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.245482 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.245483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.246256 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.246257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.246630 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.246631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.246727 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.247229 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.247237 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.248685 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.250513 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.250519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.250521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.250817 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.250994 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.250994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.250995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.251089 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.251285 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.251289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.251408 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.251487 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.251599 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":93,"threads_started":1}
I20260925 14:50:54.252286 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:54.252367 6957 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:54.252494 6957 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:54.252830 6497 ts_manager.cc:194] Registered new tserver with Master: 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:54.253238 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38015
I20260925 14:50:54.261399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 14:50:54.284725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41533
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45659,127.0.98.61:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:54.364336 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:54.364529 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:54.364550 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:54.365674 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:54.365706 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:54.365720 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:54.365732 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:54.365743 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:54.367131 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45659,127.0.98.61:41533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
W20260925 14:50:54.367223 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45659,127.0.98.61:41533'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:50:54.367323 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:54.367499 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.372387 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.372422 6962 server_base.cc:1034] running on GCE node
W20260925 14:50:54.372383 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.372413 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.372833 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.373124 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.374289 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790347854374246 us; error 59 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.376791 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 14:50:54.377162 6962 webserver.cc:460] Webserver started at http://127.0.98.61:34497/ using document root <none> and password file <none>
I20260925 14:50:54.377354 6962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.377394 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.377478 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:54.378520 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/instance:
uuid: "53f0eae7a77749c3adee8749ff80ec2a"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "7ace5e93cff8e9b2e314e3afa4247d3a"
server_key_iv: "2dd1acdf806e80552941e0042089bd0c"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.378876 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal/instance:
uuid: "53f0eae7a77749c3adee8749ff80ec2a"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "7ace5e93cff8e9b2e314e3afa4247d3a"
server_key_iv: "2dd1acdf806e80552941e0042089bd0c"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.380087 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:50:54.380726 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.380919 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:54.380996 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "53f0eae7a77749c3adee8749ff80ec2a"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "7ace5e93cff8e9b2e314e3afa4247d3a"
server_key_iv: "2dd1acdf806e80552941e0042089bd0c"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.381047 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.397352 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.399560 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:54.399667 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.403288 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41533
I20260925 14:50:54.403340 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41533 every 8 connection(s)
I20260925 14:50:54.403664 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:50:54.404601 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:54.406100 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:50:54.409175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 6962
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:54.415519 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.406787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.0.98.62:45659)
0925 14:50:54.406848 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.406851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.407044 (+ 193us) server_negotiation.cc:407] Connection header received
0925 14:50:54.407518 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.407519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.407531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.407593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.408163 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.408659 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.410065 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.410280 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.410335 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.412443 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.412444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.412445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.412453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.413952 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.414440 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.414441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.414441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.414493 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.414776 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.414777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.414778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.414963 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.415024 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.415354 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.415432 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:50:54.415691 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.406760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46318)
0925 14:50:54.406931 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.406963 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.407412 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.407662 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.407665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.407896 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.408071 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.408075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.408750 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.408751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.409976 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.409979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.410385 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.410386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.410539 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.411059 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.411067 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.412358 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.414053 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.414059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.414061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.414360 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.414559 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.414560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.414560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.414681 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.415041 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.415047 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.415204 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.415331 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.415467 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260925 14:50:54.416361 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:50:54.418437 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.409658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.61:41533)
0925 14:50:54.409819 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.409823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.409838 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:50:54.409901 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.409903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.409918 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.409981 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.410715 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.411520 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.412319 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.412763 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.412844 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.415196 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.415202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.415202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.415207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.417232 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.417596 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.417598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.417599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.417648 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.417879 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.417880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.417880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.417993 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.418160 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.418202 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.418313 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:54.423420 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.416618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48324)
0925 14:50:54.416645 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.416654 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.416687 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.416886 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.416887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.417084 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.417138 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.417139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.418041 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.418042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.418778 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.418780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.419215 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.419216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.419268 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.419556 (+ 288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.419559 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.420568 (+ 1009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.422686 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.422686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.422687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.422841 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.423004 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.423005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.423005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.423051 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.423212 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.423213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.423249 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.423297 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.423365 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260925 14:50:54.423534 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.416662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.98.61:41533)
0925 14:50:54.416782 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.416783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.416793 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:54.416805 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.416806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.416812 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.416856 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.417212 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.417973 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.418848 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.419153 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.419208 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.420640 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.420641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.420641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.420646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.422619 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.422903 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.422903 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.422904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.422951 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.423088 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.423088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.423088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.423167 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.423209 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.423317 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.423408 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260925 14:50:54.424643 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Bootstrap starting.
I20260925 14:50:54.425262 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:54.425521 7030 log.cc:824] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:54.426134 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: No bootstrap required, opened a new log
I20260925 14:50:54.427492 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } }
I20260925 14:50:54.427605 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:54.427626 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f0eae7a77749c3adee8749ff80ec2a, State: Initialized, Role: FOLLOWER
I20260925 14:50:54.427762 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } }
I20260925 14:50:54.428062 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:54.428032 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } } }
I20260925 14:50:54.428140 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:50:54.429782 7048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:54.429840 7048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:54.431285 6962 master_runner.cc:428] Detected that this master 53f0eae7a77749c3adee8749ff80ec2a is joining an existing cluster
I20260925 14:50:54.431322 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41533
I20260925 14:50:54.439065 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.432234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46334)
0925 14:50:54.432376 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.432385 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.432441 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.432566 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.432566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.432741 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.432805 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.432806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.433498 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.433499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.434166 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.434168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.434696 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.434697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.434752 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.435594 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.435598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.435874 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.437841 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.437842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.437842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.438064 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.438292 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.438292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.438292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.438339 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.438673 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.438674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.438716 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.438838 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.438956 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260925 14:50:54.439338 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.432264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.0.98.62:45659)
0925 14:50:54.432310 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.432312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.432429 (+ 117us) server_negotiation.cc:407] Connection header received
0925 14:50:54.432500 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.432501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.432511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.432559 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.432884 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.433419 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.434236 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.434540 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.434616 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.435963 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.435964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.435965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.435973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.437749 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.438151 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.438152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.438153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.438209 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.438444 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.438444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.438445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.438581 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.438646 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.438866 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.438945 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:54.441514 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.432366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48336)
0925 14:50:54.434432 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.434443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.434504 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.434651 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.434652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.434838 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.434893 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.434894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.435581 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.435582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.436410 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.436413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.436949 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.436950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.437006 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.437694 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.437700 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.438128 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.440136 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.440137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.440137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.440637 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.440824 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.440825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.440826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.440907 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.441168 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.441168 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.441205 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.441274 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.441372 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2049,"thread_start_us":67,"threads_started":1}
I20260925 14:50:54.441531 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.432315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.0.98.61:41533)
0925 14:50:54.432351 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.432356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.434506 (+ 2150us) server_negotiation.cc:407] Connection header received
0925 14:50:54.434560 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.434560 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.434583 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.434633 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.434970 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.435494 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.436496 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.436826 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.436923 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.438207 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.438208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.438208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.438215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.440054 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.440705 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.440706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.440707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.440767 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.440968 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.440968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.440969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.441091 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.441175 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.441298 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.441423 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":17}
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:54.449204 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.441615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.61:41533)
0925 14:50:54.441648 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.441651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.442295 (+ 644us) server_negotiation.cc:407] Connection header received
0925 14:50:54.442323 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.442323 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.442331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.442529 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.443006 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.443428 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.444498 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.444656 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.444701 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.446684 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.446685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.446686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.446692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.448021 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.448417 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.448418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.448419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.448470 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.448673 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.448673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.448674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.448780 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.448849 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.449056 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.449125 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:54.449251 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.441431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48346)
0925 14:50:54.442142 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.442192 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.442259 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.442549 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.442554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.442878 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.442932 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.442934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.443501 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.443502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.444423 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.444425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.444721 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.444722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.444800 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.445184 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.445190 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.446598 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.448104 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.448111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.448113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.448335 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.448540 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.448541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.448541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.448601 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.448854 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.448861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.448910 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.449031 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.449140 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":64,"threads_started":1}
I20260925 14:50:54.449530 7037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:54.449607 7037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } }
I20260925 14:50:54.449811 7037 leader_election.cc:288] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659)
I20260925 14:50:54.450089 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41533: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41533
}
attrs {
promote: true
}
}
dest_uuid: "517c35e223d647748517bc8379274670"
cas_config_opid_index: -1
I20260925 14:50:54.450402 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } }
I20260925 14:50:54.450860 6508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53f0eae7a77749c3adee8749ff80ec2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "517c35e223d647748517bc8379274670" is_pre_election: true
I20260925 14:50:54.450940 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } }
I20260925 14:50:54.451166 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:54.451184 6980 leader_election.cc:302] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a; no voters: 517c35e223d647748517bc8379274670
I20260925 14:50:54.451870 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41533
I20260925 14:50:54.451952 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } }
I20260925 14:50:54.452023 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:54.452062 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:50:54.452081 6962 master.cc:537] Master@127.0.98.61:41533 shutting down...
I20260925 14:50:54.452811 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:50:54.453488 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 14:50:54.455257 7037 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:50:54.455967 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:54.456040 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:54.456087 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: stopping tablet replica
I20260925 14:50:54.466701 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.466571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48354)
0925 14:50:54.466629 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.466669 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:50:54.466815 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 -> Peer 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61:41533): Couldn't send request to peer 53f0eae7a77749c3adee8749ff80ec2a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:50:54.468674 6962 master.cc:552] Master@127.0.98.61:41533 shutdown complete.
I20260925 14:50:54.468736 6962 master_runner.cc:305] Clearing existing system tablet
I20260925 14:50:54.468876 6962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.468940 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.469770 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:54.472260 7062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.472373 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.472421 6962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260925 14:50:54.472445 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "53f0eae7a77749c3adee8749ff80ec2a"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "7ace5e93cff8e9b2e314e3afa4247d3a"
server_key_iv: "2dd1acdf806e80552941e0042089bd0c"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.473974 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:54.475653 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:50:54.475718 6962 log.cc:1196] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:50:54.475935 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Deleting consensus metadata
I20260925 14:50:54.476063 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:45659
I20260925 14:50:54.476647 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45659
I20260925 14:50:54.482991 7068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.476947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46338)
0925 14:50:54.477049 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.477059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.477123 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.477263 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.477263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.477407 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.477454 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.477456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.478501 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.478502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.479053 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.479054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.479521 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.479522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.479568 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.480052 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.480056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.480360 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.482045 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.482047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.482047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.482207 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.482456 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.482457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.482457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.482510 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.482761 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.482762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.482795 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.482845 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.482918 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":66,"threads_started":1}
I20260925 14:50:54.483033 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.477007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.0.98.62:45659)
0925 14:50:54.477053 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.477056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.477103 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:50:54.477189 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.477189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.477200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.477261 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.477520 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.478298 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.479135 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.479447 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.479519 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.480444 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.480446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.480446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.480453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.481965 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.482304 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.482305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.482306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.482390 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.482569 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.482570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.482570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.482699 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.482768 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.482867 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.482940 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:50:54.483575 6518 tablet_copy_service.cc:140] P 517c35e223d647748517bc8379274670: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 53f0eae7a77749c3adee8749ff80ec2a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46338)
I20260925 14:50:54.483662 6518 tablet_copy_service.cc:161] P 517c35e223d647748517bc8379274670: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 53f0eae7a77749c3adee8749ff80ec2a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46338: session id = 53f0eae7a77749c3adee8749ff80ec2a-00000000000000000000000000000000
I20260925 14:50:54.484341 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:50:54.484921 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:50:54.485767 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: tablet copy: Starting download of 0 data blocks...
I20260925 14:50:54.485930 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: tablet copy: Starting download of 1 WAL segments...
I20260925 14:50:54.486785 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:50:54.487511 6518 tablet_copy_service.cc:342] P 517c35e223d647748517bc8379274670: Request end of tablet copy session 53f0eae7a77749c3adee8749ff80ec2a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:46338
I20260925 14:50:54.487565 6518 tablet_copy_service.cc:433] P 517c35e223d647748517bc8379274670: ending tablet copy session 53f0eae7a77749c3adee8749ff80ec2a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 53f0eae7a77749c3adee8749ff80ec2a
I20260925 14:50:54.489898 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.491008 7069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.491109 7072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.491125 7070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.491297 6962 server_base.cc:1034] running on GCE node
I20260925 14:50:54.491379 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.491520 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.492648 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790347854492639 us; error 24 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.493937 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 14:50:54.494215 6962 webserver.cc:460] Webserver started at http://127.0.98.61:41627/ using document root <none> and password file <none>
I20260925 14:50:54.494330 6962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.494405 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.494992 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.495501 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.495633 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:50:54.495676 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "53f0eae7a77749c3adee8749ff80ec2a"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "7ace5e93cff8e9b2e314e3afa4247d3a"
server_key_iv: "2dd1acdf806e80552941e0042089bd0c"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.495731 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.505796 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.507333 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.509950 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41533
I20260925 14:50:54.510207 7131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41533 every 8 connection(s)
I20260925 14:50:54.510380 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:50:54.511202 7132 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:50:54.511734 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Bootstrap starting.
I20260925 14:50:54.514559 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:50:54.514813 7132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: Bootstrap complete.
I20260925 14:50:54.515009 7132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } }
I20260925 14:50:54.515098 7132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 53f0eae7a77749c3adee8749ff80ec2a, State: Initialized, Role: LEARNER
I20260925 14:50:54.515137 7132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } }
I20260925 14:50:54.515319 7132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:54.515311 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } }
I20260925 14:50:54.515446 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: LEARNER
I20260925 14:50:54.516830 7146 catalog_manager.cc:1259] Loaded cluster ID: 30232186eaf64c3a811ff2153377b04d
I20260925 14:50:54.516885 7146 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: loading cluster ID for follower catalog manager: success
I20260925 14:50:54.518018 7146 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: acquiring CA information for follower catalog manager: success
I20260925 14:50:54.518316 7146 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:50:54.526865 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.520356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.61:41533)
0925 14:50:54.520453 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.520455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.520463 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:54.520484 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.520485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.520501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.520570 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.521106 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.521758 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.522521 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.522747 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.522823 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.524251 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.524253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.524253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.524266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.525665 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.526012 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.526013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.526014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.526068 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.526270 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.526271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.526271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.526445 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.526570 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.526664 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.526750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 14:50:54.553151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32957
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:54.636576 7150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:54.636790 7150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:54.636811 7150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:54.638031 7150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:54.638072 7150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:54.638087 7150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:54.638123 7150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:54.638149 7150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:54.639693 7150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:54.640076 7150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:54.640352 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.645251 7158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.645316 7155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.645481 7156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.645767 7150 server_base.cc:1034] running on GCE node
I20260925 14:50:54.646023 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.646317 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.647529 7150 hybrid_clock.cc:648] HybridClock initialized: now 1790347854647513 us; error 31 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.650167 7150 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 14:50:54.650666 7150 webserver.cc:460] Webserver started at http://127.0.98.60:35129/ using document root <none> and password file <none>
I20260925 14:50:54.650914 7150 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.650957 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.651121 7150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:50:54.652153 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/instance:
uuid: "7292d19b964a481ab166a67930a02af4"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "00c9bca1e009ab7cfccee6e192569834"
server_key_iv: "838ebee7a06b2dfdc32248da12b0e233"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.652506 7150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal/instance:
uuid: "7292d19b964a481ab166a67930a02af4"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "00c9bca1e009ab7cfccee6e192569834"
server_key_iv: "838ebee7a06b2dfdc32248da12b0e233"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.653846 7150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:50:54.654613 7165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.654801 7150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:54.654894 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "7292d19b964a481ab166a67930a02af4"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "00c9bca1e009ab7cfccee6e192569834"
server_key_iv: "838ebee7a06b2dfdc32248da12b0e233"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.654974 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.664700 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.666677 7150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:54.666823 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.670946 7150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32957
I20260925 14:50:54.670977 7217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32957 every 8 connection(s)
I20260925 14:50:54.671314 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:50:54.672082 7218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:54.673552 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:50:54.677961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7150
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:54.683627 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.674416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.0.98.62:45659)
0925 14:50:54.674479 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.674481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.675030 (+ 549us) server_negotiation.cc:407] Connection header received
0925 14:50:54.675563 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.675564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.675577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.675641 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.676215 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.676762 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.677801 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.677984 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.678068 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.680572 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.680574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.680574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.680582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.682079 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.682593 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.682594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.682595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.682649 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.682958 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.682959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.682959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.683057 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.683104 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.683451 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.683537 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:54.683696 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.674287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46350)
0925 14:50:54.674533 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.674566 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.675461 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.675709 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.675712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.675945 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.676122 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.676125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.676873 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.676875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.677707 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.677709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.678073 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.678075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.678237 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.678897 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.678905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.680482 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.682168 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.682172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.682174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.682519 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.682722 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.682723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.682723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.682886 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.683118 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.683122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.683275 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.683389 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.683519 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":130,"threads_started":1}
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:50:54.684247 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:50:54.687400 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.678589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.0.98.60:32957)
0925 14:50:54.678776 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.678780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.679124 (+ 344us) server_negotiation.cc:407] Connection header received
0925 14:50:54.679148 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.679150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.679171 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.679248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.679962 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.680657 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.681366 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.681658 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.681738 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.684547 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.684553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.684553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.684560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.686175 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.686589 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.686592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.686592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.686650 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.686850 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.686851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.686852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.687008 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.687170 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.687213 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.687292 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:54.690305 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.684499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48372)
0925 14:50:54.684539 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.684548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.684582 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.684722 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.684722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.684830 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.684868 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.684869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.685536 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.685537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.686036 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.686037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.686433 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.686433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.686505 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.686879 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.686884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.688126 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.689637 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.689637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.689638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.689784 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.689928 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.689929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.689929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.689967 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.690118 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.690118 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.690145 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.690188 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.690252 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:50:54.690332 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.684556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.0.98.61:41533)
0925 14:50:54.684604 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.684606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.684615 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:54.684641 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.684642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.684657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.684721 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.684948 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.685421 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.686119 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.686330 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.686434 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.688202 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.688203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.688203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.688210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.689572 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.689845 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.689846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.689846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.689890 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.690004 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.690004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.690004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.690086 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.690126 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.690203 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.690259 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:54.690699 7218 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:50:54.696373 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.690923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46850)
0925 14:50:54.690969 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.690978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.691015 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.691133 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.691134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.691277 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.691298 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.691299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.691895 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.691896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.692363 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.692364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.692715 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.692716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.692760 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.693079 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.693083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.694096 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.695411 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.695412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.695413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.695601 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.695788 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.695789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.695789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.695843 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.696072 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.696073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.696106 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.696173 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.696266 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:50:54.696425 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.690981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.0.98.60:32957)
0925 14:50:54.691011 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.691013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.691021 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:54.691070 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.691071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.691079 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.691130 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.691371 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.691808 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.692439 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.692637 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.692723 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.694185 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.694186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.694186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.694192 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.695320 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.695664 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.695665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.695665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.695725 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.695913 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.695914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.695914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.696012 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.696080 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.696200 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.696312 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":16}
I20260925 14:50:54.697633 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Bootstrap starting.
I20260925 14:50:54.698465 7218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:54.698695 7218 log.cc:824] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:54.699353 7218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: No bootstrap required, opened a new log
I20260925 14:50:54.700649 7218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } }
I20260925 14:50:54.700773 7218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:54.700796 7218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7292d19b964a481ab166a67930a02af4, State: Initialized, Role: FOLLOWER
I20260925 14:50:54.700913 7218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } }
I20260925 14:50:54.701161 7218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:54.701164 7223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } } }
I20260925 14:50:54.701293 7223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:50:54.703068 7234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:50:54.703140 7234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:50:54.705204 7150 master_runner.cc:428] Detected that this master 7292d19b964a481ab166a67930a02af4 is joining an existing cluster
I20260925 14:50:54.705251 7150 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:32957
I20260925 14:50:54.712925 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46366)
0925 14:50:54.706440 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.706449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.706522 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.706717 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.706717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.706933 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.707010 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.707011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.707975 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.707976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.708950 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.708952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709473 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.709536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.710030 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.710035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.710408 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.712054 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.712055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.712055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.712210 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.712405 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.712406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.712406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.712453 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.712698 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.712699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.712740 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.712789 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.712857 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":51,"thread_start_us":80,"threads_started":1}
I20260925 14:50:54.713007 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.0.98.62:45659)
0925 14:50:54.706333 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.706335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.706513 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:50:54.706608 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.706609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.706620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.706651 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.707102 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.707877 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709050 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709348 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709385 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.710479 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.710481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.710481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.710488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.711968 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.712278 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.712279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.712279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.712340 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.712509 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.712510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.712510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.712637 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.712714 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.712816 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.712903 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347853, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:50:54.730108 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.0.98.60:32957)
0925 14:50:54.706720 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.706722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.706905 (+ 183us) server_negotiation.cc:407] Connection header received
0925 14:50:54.706984 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.706985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.706992 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.707017 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.707388 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.708077 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709104 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709373 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709410 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.727263 (+ 17853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.727264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.727264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.727270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.728962 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729368 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.729369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.729369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.729422 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729588 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.729588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.729588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.729701 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.729775 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.729943 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.730016 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:54.730832 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.98.61:41533)
0925 14:50:54.706406 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.706408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.706642 (+ 234us) server_negotiation.cc:407] Connection header received
0925 14:50:54.706736 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.706737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.706747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.706777 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.707164 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.707884 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709050 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709345 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709385 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.727546 (+ 18161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.727548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.727548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.727556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.729271 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729964 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.729965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.729966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.730022 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.730227 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.730228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.730229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.730381 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.730441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.730660 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.730738 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 14:50:54.730850 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46852)
0925 14:50:54.706853 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.706862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.706924 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.707074 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.707075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.707255 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.707311 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.707313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.708171 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.708172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.709014 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.709016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709473 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.709526 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.726726 (+ 17200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.726731 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.727170 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.729053 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.729054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.729291 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.729483 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.729483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.729529 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.729813 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.729814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.729847 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.730220 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.730743 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260925 14:50:54.731069 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.706452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48386)
0925 14:50:54.706585 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.706595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.706664 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.706860 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.706860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.707020 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.707088 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.707089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.707975 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.707976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.708950 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.708952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.709473 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.709474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.709536 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.726882 (+ 17346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.726887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.727438 (+ 551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.729368 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.729369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.729369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.729873 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.730095 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.730095 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.730096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.730155 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.730468 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.730469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.730511 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.730577 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.730989 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":362,"thread_start_us":90,"threads_started":1}
I20260925 14:50:54.731801 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.714882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46862)
0925 14:50:54.714932 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.714941 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.714976 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.715351 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.715352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.715513 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.715544 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.715545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.716381 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.716382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.717004 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.717006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.717387 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.717388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.717454 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.717864 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.717869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.719226 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.730699 (+ 11473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.730700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.730701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.730922 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.731176 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.731177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.731177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.731236 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.731487 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.731487 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.731529 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.731577 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.731687 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:50:54.732182 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.715040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.0.98.60:32957)
0925 14:50:54.715173 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.715174 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.715181 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:50:54.715244 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.715245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.715266 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.715347 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.715625 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.716264 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.717079 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.717305 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.717385 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.719329 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.719330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.719330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.719337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.730622 (+ 11285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.731047 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.731048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.731049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.731102 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.731303 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.731304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.731304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.731413 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.731976 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.732016 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.732088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260925 14:50:54.732597 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:32957: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7292d19b964a481ab166a67930a02af4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 32957
}
attrs {
promote: true
}
}
dest_uuid: "517c35e223d647748517bc8379274670"
cas_config_opid_index: 5
I20260925 14:50:54.732848 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } }
I20260925 14:50:54.733201 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7292d19b964a481ab166a67930a02af4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.733476 7192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:54.734220 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:32957
I20260925 14:50:54.734292 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.734373 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:54.734332 7192 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:50:54.734659 7150 master.cc:537] Master@127.0.98.60:32957 shutting down...
I20260925 14:50:54.735013 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:50:54.735673 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 -> Peer 7292d19b964a481ab166a67930a02af4 (127.0.98.60:32957): Couldn't send request to peer 7292d19b964a481ab166a67930a02af4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:50:54.737506 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.733322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48394)
0925 14:50:54.733474 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.733483 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.733523 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.733720 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.733721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.733723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:54.733758 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.733759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.734739 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.734740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.735930 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.735934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.736699 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.736700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.737295 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.737316 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.737379 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.737423 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260925 14:50:54.737736 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.733414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.61:41533)
0925 14:50:54.733447 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.733448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.733572 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:50:54.733592 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.733592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.733612 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.733664 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:54.733823 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.734615 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.736031 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.736601 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.736669 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.736746 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.737640 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.737681 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:54.737767 7150 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:54.737874 7150 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:54.737929 7150 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: stopping tablet replica
I20260925 14:50:54.737927 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 LEARNER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:50:54.738179 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:50:54.738775 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 7292d19b964a481ab166a67930a02af4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.739863 7135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517c35e223d647748517bc8379274670. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.739959 7135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: LEARNER
I20260925 14:50:54.739806 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: true } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.740142 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: LEARNER
I20260925 14:50:54.740424 7057 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: attempting to promote NON_VOTER 53f0eae7a77749c3adee8749ff80ec2a to VOTER
I20260925 14:50:54.740623 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } }
I20260925 14:50:54.741274 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 LEARNER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260925 14:50:54.741523 6451 connection.cc:535] client connection to 127.0.98.60:32957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 14:50:54.741597 6485 connection.cc:535] client connection to 127.0.98.60:32957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:50:54.741786 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:50:54.741880 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.741793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46876)
0925 14:50:54.741827 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.741852 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 14:50:54.741958 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 -> Peer 7292d19b964a481ab166a67930a02af4 (127.0.98.60:32957): Couldn't send request to peer 7292d19b964a481ab166a67930a02af4. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32957: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:50:54.742401 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.742542 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.743271 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.743376 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.743463 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:54.743386 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:54.743746 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.743819 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:50:54.752597 7150 master.cc:552] Master@127.0.98.60:32957 shutdown complete.
I20260925 14:50:54.752653 7150 master_runner.cc:305] Clearing existing system tablet
I20260925 14:50:54.752776 7150 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.752842 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.753513 7150 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:54.753979 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.754107 7150 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.754166 7150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.754199 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "7292d19b964a481ab166a67930a02af4"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "00c9bca1e009ab7cfccee6e192569834"
server_key_iv: "838ebee7a06b2dfdc32248da12b0e233"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.754863 7150 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:54.755991 7150 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:50:54.756045 7150 log.cc:1196] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:50:54.756225 7150 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Deleting consensus metadata
I20260925 14:50:54.756353 7150 master_runner.cc:315] Copying system tablet from 127.0.98.62:45659
I20260925 14:50:54.756956 7150 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45659
I20260925 14:50:54.763537 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.757377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.0.98.62:45659)
0925 14:50:54.757422 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.757424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.757474 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:50:54.757540 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.757541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.757549 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.757601 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.757979 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.758524 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.759546 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.759792 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.759860 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.760974 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.760975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.760975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.760983 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.762548 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.762909 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.762910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.762910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.762959 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.763135 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.763136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.763136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.763229 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.763276 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.763423 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.763478 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:50:54.763625 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.757298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46374)
0925 14:50:54.757414 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.757424 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.757480 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.757617 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.757617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.757832 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.757904 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.757905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.758593 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.758594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.759470 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.759472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.759869 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.759870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.759939 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.760491 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.760497 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.760880 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.762618 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.762620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.762620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.762843 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.763016 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.763017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.763017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.763079 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.763286 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.763287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.763332 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.763408 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.763505 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260925 14:50:54.763859 6518 tablet_copy_service.cc:140] P 517c35e223d647748517bc8379274670: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7292d19b964a481ab166a67930a02af4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46374)
I20260925 14:50:54.763918 6518 tablet_copy_service.cc:161] P 517c35e223d647748517bc8379274670: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7292d19b964a481ab166a67930a02af4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46374: session id = 7292d19b964a481ab166a67930a02af4-00000000000000000000000000000000
I20260925 14:50:54.764348 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:50:54.764809 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:50:54.765657 7150 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: tablet copy: Starting download of 0 data blocks...
I20260925 14:50:54.765811 7150 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: tablet copy: Starting download of 1 WAL segments...
I20260925 14:50:54.766536 7150 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:50:54.767274 6518 tablet_copy_service.cc:342] P 517c35e223d647748517bc8379274670: Request end of tablet copy session 7292d19b964a481ab166a67930a02af4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:46374
I20260925 14:50:54.767421 6518 tablet_copy_service.cc:433] P 517c35e223d647748517bc8379274670: ending tablet copy session 7292d19b964a481ab166a67930a02af4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7292d19b964a481ab166a67930a02af4
I20260925 14:50:54.769702 7150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.770849 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.770917 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.770998 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.771059 7150 server_base.cc:1034] running on GCE node
I20260925 14:50:54.771306 7150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.771437 7150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.772559 7150 hybrid_clock.cc:648] HybridClock initialized: now 1790347854772552 us; error 23 us; skew 500 ppm
I20260925 14:50:54.773135 6689 heartbeater.cc:503] Master 127.0.98.62:45659 was elected leader, sending a full tablet report...
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.773952 7150 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 14:50:54.774210 7150 webserver.cc:460] Webserver started at http://127.0.98.60:38097/ using document root <none> and password file <none>
I20260925 14:50:54.774365 7150 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.774402 7150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.775029 7150 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.775502 7265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.775620 7150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.775689 7150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "7292d19b964a481ab166a67930a02af4"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "00c9bca1e009ab7cfccee6e192569834"
server_key_iv: "838ebee7a06b2dfdc32248da12b0e233"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.775748 7150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.780180 7150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.781864 7150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.784878 7150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32957
I20260925 14:50:54.785130 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32957 every 8 connection(s)
I20260925 14:50:54.785252 7150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:50:54.785863 7318 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:50:54.786252 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Bootstrap starting.
I20260925 14:50:54.788512 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:50:54.788714 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Bootstrap complete.
I20260925 14:50:54.788930 7318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } }
I20260925 14:50:54.789004 7318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7292d19b964a481ab166a67930a02af4, State: Initialized, Role: LEARNER
I20260925 14:50:54.789031 7318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } }
I20260925 14:50:54.789225 7318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:54.789234 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:54.789361 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:50:54.790664 7332 catalog_manager.cc:1259] Loaded cluster ID: 30232186eaf64c3a811ff2153377b04d
I20260925 14:50:54.790711 7332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: loading cluster ID for follower catalog manager: success
I20260925 14:50:54.792338 7332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: acquiring CA information for follower catalog manager: success
I20260925 14:50:54.792609 7332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:50:54.799309 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.792192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.0.98.60:32957)
0925 14:50:54.792300 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.792302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.792313 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:50:54.792336 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.792337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.792348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.793152 (+ 804us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.793185 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.794150 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.794912 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.795215 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.795327 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.796598 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.796599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.796600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.796607 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.798188 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.798567 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.798568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.798568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.798632 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.798816 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.798817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.798817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.798943 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.799045 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.799137 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.799220 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":47,"threads_started":1}
I20260925 14:50:54.799813 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6560
I20260925 14:50:54.806950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32957
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45659
--webserver_port=42673
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:54.889509 7334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:54.889716 7334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:54.889755 7334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:54.891083 7334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:54.891148 7334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:54.891191 7334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:54.892613 7334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42673
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:54.892879 7334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:54.893110 7334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:54.898049 7340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.898085 7342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:54.898053 7339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:54.898264 7334 server_base.cc:1034] running on GCE node
I20260925 14:50:54.898489 7334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:54.898762 7334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:54.899941 7334 hybrid_clock.cc:648] HybridClock initialized: now 1790347854899922 us; error 41 us; skew 500 ppm
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:54.902582 7334 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:50:54.902997 7334 webserver.cc:460] Webserver started at http://127.0.98.1:42673/ using document root <none> and password file <none>
I20260925 14:50:54.903226 7334 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:54.903308 7334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:54.904565 7334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:54.905126 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.905305 7334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.905407 7334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4c862c5525d4478496446ad8ecf53d52"
format_stamp: "Formatted at 2026-09-25 14:50:53 on dist-test-slave-gpc1"
server_key: "f4268524572f298dfc88a3ab4ec792fd"
server_key_iv: "652bf6233a201daea0f426245e48df39"
server_key_version: "encryptionkey@0"
I20260925 14:50:54.905740 7334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:54.924995 7334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:54.926882 7334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:54.927037 7334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:54.927248 7334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:54.927541 7334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:54.927596 7334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.927637 7334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:54.927665 7334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:54.933719 7334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32957
I20260925 14:50:54.933773 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32957 every 8 connection(s)
I20260925 14:50:54.934162 7334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:54.943017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7334
I20260925 14:50:54.943181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6694
Sep 25 14:50:54 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347854, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:50:54.947269 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.935312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33883 (local address 127.0.98.61:41533)
0925 14:50:54.935371 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.935373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.935409 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:50:54.936694 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.936695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.936708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.936768 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.937546 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938028 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939392 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939625 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939692 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.942739 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.942741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.942741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.942749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.945825 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946179 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.946180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.946181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.946231 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946558 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.946559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.946559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.946712 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.946753 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.947099 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.947176 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:50:54.947273 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.934894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48811 (local address 127.0.98.60:32957)
0925 14:50:54.934935 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.934937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.935203 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:50:54.936721 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.936723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.936737 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.936767 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.937549 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938298 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939338 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939640 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939696 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.944014 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.944015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.944016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.944024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.945634 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946148 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.946149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.946150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.946219 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946559 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.946559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.946560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.946686 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.946730 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.947099 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.947177 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:50:54.947758 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.934946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47271 (local address 127.0.98.62:45659)
0925 14:50:54.934985 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:54.934986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:54.935358 (+ 372us) server_negotiation.cc:407] Connection header received
0925 14:50:54.936711 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:54.936712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:54.936725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:54.936759 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:54.937548 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938308 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939206 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939511 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939575 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.942138 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:54.942139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:54.942140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:54.942146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:54.944508 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946846 (+ 2338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.946848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.946848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.946907 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:54.947103 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:54.947104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:54.947104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:54.947237 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:54.947276 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:54.947538 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:50:54.947623 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:54.947822 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.935188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.98.1:33883)
0925 14:50:54.935341 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.935352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.936598 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.936856 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.936858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.937225 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.937437 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.937441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938128 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.938129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939289 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.939291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939697 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939765 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.940428 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.940432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.942641 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.945902 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.945903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.945904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.946101 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.946301 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.946302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.946475 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.946798 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.946798 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.946937 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.947408 (+ 471us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.947592 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":16,"thread_start_us":60,"threads_started":1}
I20260925 14:50:54.948310 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.934914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.98.1:48811)
0925 14:50:54.935077 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.935110 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.936581 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.936853 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.936857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.937227 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.937438 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.937444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938403 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.938403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939246 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.939249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939741 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939823 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.940363 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.940371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.943924 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.945731 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.945737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.945739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.946057 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.946299 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.946300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.946475 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.946780 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.946786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.946937 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.947716 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.948189 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":39,"thread_start_us":87,"threads_started":1}
I20260925 14:50:54.948395 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:54.934914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:47271)
0925 14:50:54.935294 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:54.935303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:54.936587 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:54.936852 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:54.936857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:54.937231 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:54.937438 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.937444 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.938425 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.938426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939091 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:54.939093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:54.939590 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:54.939591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:54.939738 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:54.940489 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:54.940494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:54.942035 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:54.946522 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.946524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.946754 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.946979 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:54.946979 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:54.946980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:54.947039 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:54.947317 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:54.947318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:54.947426 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:50:54.947881 (+ 455us) client_negotiation.cc:240] Negotiation successful
0925 14:50:54.948251 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":319,"thread_start_us":90,"threads_started":1}
I20260925 14:50:54.949167 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41533
I20260925 14:50:54.949205 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:54.949373 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32957
I20260925 14:50:54.949406 7464 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:54.949420 7463 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:54.949452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37901
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45659
--webserver_port=43645
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:50:54.949615 7463 heartbeater.cc:511] Master 127.0.98.60:32957 requested a full tablet report, sending...
I20260925 14:50:54.949613 7464 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:54.949295 7465 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:54.949851 7465 heartbeater.cc:511] Master 127.0.98.61:41533 requested a full tablet report, sending...
I20260925 14:50:54.950043 6497 ts_manager.cc:194] Re-registered known tserver with Master: 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
I20260925 14:50:54.950570 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47271
I20260925 14:50:54.950954 7096 ts_manager.cc:194] Registered new tserver with Master: 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
I20260925 14:50:54.950977 7282 ts_manager.cc:194] Registered new tserver with Master: 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
W20260925 14:50:55.031525 7470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:55.031770 7470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:55.031813 7470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:55.033128 7470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:55.033210 7470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:55.033260 7470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:55.034818 7470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43645
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:55.035096 7470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:55.035336 7470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:55.040503 7475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:55.040619 7476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:55.040522 7478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:55.040706 7470 server_base.cc:1034] running on GCE node
I20260925 14:50:55.040962 7470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:55.041200 7470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:55.042435 7470 hybrid_clock.cc:648] HybridClock initialized: now 1790347855042408 us; error 34 us; skew 500 ppm
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:55.045014 7470 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:50:55.045465 7470 webserver.cc:460] Webserver started at http://127.0.98.2:43645/ using document root <none> and password file <none>
I20260925 14:50:55.045689 7470 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:55.045780 7470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:55.047181 7470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:55.047756 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.047966 7470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:55.048069 7470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "13d9857b70164976ae22a6faf92b58cc"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "9c38c14b6079e97375534255a0f588f6"
server_key_iv: "90c70973dc3d33c59c876477122f5339"
server_key_version: "encryptionkey@0"
I20260925 14:50:55.048405 7470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:55.064819 7470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:55.066808 7470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:55.066974 7470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:55.067195 7470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:55.067512 7470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:55.067570 7470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.067611 7470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:55.067641 7470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.073311 7470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37901
I20260925 14:50:55.073344 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37901 every 8 connection(s)
I20260925 14:50:55.073813 7470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:55.074441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7470
I20260925 14:50:55.074605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6828
I20260925 14:50:55.081430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35201
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45659
--webserver_port=42753
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:50:55.086534 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.074481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53873 (local address 127.0.98.60:32957)
0925 14:50:55.074525 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.074527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.075378 (+ 851us) server_negotiation.cc:407] Connection header received
0925 14:50:55.076068 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.076069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.076082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.076131 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.077045 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.077816 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079400 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079723 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079767 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.082941 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.082942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.082943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.082951 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.084683 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085339 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.085340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.085341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.085398 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085685 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.085686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.085686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.085843 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.085886 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.086307 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.086420 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:50:55.087062 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.074507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:60871)
0925 14:50:55.075224 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.075269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.075959 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.077203 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.077204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.077378 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.077432 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.077434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.078197 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.078198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.079300 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.079303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079893 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.079995 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.080982 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.080987 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.083028 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.085890 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.085891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.086115 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.086337 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.086359 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.086359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.086418 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.086692 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.086693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.086728 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.086782 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.086853 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":6,"spinlock_wait_cycles":384,"thread_start_us":130,"threads_started":1}
I20260925 14:50:55.087128 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.078439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48589 (local address 127.0.98.61:41533)
0925 14:50:55.078482 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.078484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.078493 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:55.078516 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.078517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.078526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.078595 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.078912 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079630 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.080609 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.080823 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.080910 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.083570 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.083572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.083572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.083579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.085448 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085933 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.085934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.085935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.085994 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.086200 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.086201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.086201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.086334 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.086419 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.086910 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.086995 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:50:55.087060 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.074508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.98.2:53873)
0925 14:50:55.075324 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.075333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.075959 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.076267 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.076272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.076699 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.076952 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.076958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.077938 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.077939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.079303 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.079306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079853 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.079995 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.080575 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.080585 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.082832 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.084930 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.084936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.084938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.085242 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.085471 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.085472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.085605 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.085934 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.085940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.086149 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.086635 (+ 486us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.086817 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":98,"threads_started":1}
I20260925 14:50:55.087332 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.075143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60871 (local address 127.0.98.62:45659)
0925 14:50:55.075245 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.075247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.075355 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:50:55.077098 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.077099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.077113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.077143 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.077515 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.078126 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079386 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079677 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079729 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.083124 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.083125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.083126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.083133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.085806 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.086196 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.086197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.086197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.086256 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.086485 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.086485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.086486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.086608 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.086667 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.087132 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.087214 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 14:50:55.087594 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32957
I20260925 14:50:55.087594 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:55.087711 7599 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.087711 7600 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.087913 7600 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:55.087913 7599 heartbeater.cc:511] Master 127.0.98.60:32957 requested a full tablet report, sending...
I20260925 14:50:55.088323 6497 ts_manager.cc:194] Re-registered known tserver with Master: 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901)
I20260925 14:50:55.088392 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.074794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.98.2:48589)
0925 14:50:55.075511 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.075520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.078393 (+ 2873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.078611 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.078612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.078769 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.078819 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.078820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.079742 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.079742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.080531 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.080533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.080912 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.080913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.080974 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.081355 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.081360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.083477 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.085546 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.085548 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.085548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.085840 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.086066 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.086067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.086067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.086128 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.086473 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.086474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.086531 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.087933 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.088028 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":54,"threads_started":1}
I20260925 14:50:55.088392 7282 ts_manager.cc:194] Registered new tserver with Master: 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901)
I20260925 14:50:55.088820 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41533
I20260925 14:50:55.088871 7601 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.088852 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60871
I20260925 14:50:55.088953 7601 heartbeater.cc:511] Master 127.0.98.61:41533 requested a full tablet report, sending...
I20260925 14:50:55.089216 7096 ts_manager.cc:194] Registered new tserver with Master: 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901)
W20260925 14:50:55.164253 7606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:55.164450 7606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:55.164474 7606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:55.165515 7606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:55.165547 7606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:55.165584 7606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:55.166999 7606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42753
--tserver_master_addrs=127.0.98.62:45659,127.0.98.61:41533,127.0.98.60:32957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:50:55.167168 7606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:55.167341 7606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:55.172253 7611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:55.172261 7612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:55.172405 7614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:55.172554 7606 server_base.cc:1034] running on GCE node
I20260925 14:50:55.172739 7606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:55.172989 7606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:55.174111 7606 hybrid_clock.cc:648] HybridClock initialized: now 1790347855174085 us; error 41 us; skew 500 ppm
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:55.176934 7606 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:50:55.177356 7606 webserver.cc:460] Webserver started at http://127.0.98.3:42753/ using document root <none> and password file <none>
I20260925 14:50:55.177614 7606 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:55.177696 7606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:55.179064 7606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:55.179622 7621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.179805 7606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.179873 7606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "394d329381b2487283d737bbcc863193"
format_stamp: "Formatted at 2026-09-25 14:50:54 on dist-test-slave-gpc1"
server_key: "3296411fa1ae286832393697a8535bad"
server_key_iv: "18272fcfd699c98ee7a57d02d438187b"
server_key_version: "encryptionkey@0"
I20260925 14:50:55.180209 7606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:55.193455 7606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:55.195511 7606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:55.195650 7606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:55.195919 7606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:55.196213 7606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:55.196265 7606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.196307 7606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:55.196336 7606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:55.202138 7606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35201
I20260925 14:50:55.202188 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35201 every 8 connection(s)
I20260925 14:50:55.202531 7606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:55.206768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7606
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 25 14:50:55 dist-test-slave-gpc1 krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347855, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:50:55.214439 7147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56161 (local address 127.0.98.61:41533)
0925 14:50:55.203362 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.203365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.203554 (+ 189us) server_negotiation.cc:407] Connection header received
0925 14:50:55.204372 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.204373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.204382 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.204436 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.205073 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205823 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.206987 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207287 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207358 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.211287 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.211287 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.211288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.211294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.212811 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.213306 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.213307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.213307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.213395 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.213704 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.213704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.213704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.213830 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.213892 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.214254 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.214322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:50:55.214655 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.98.3:56161)
0925 14:50:55.203419 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.203459 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.204102 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.204460 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.204462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.204811 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.204961 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.204965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205921 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.205922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.206900 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.206902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207416 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.207491 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.208102 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.208112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.211196 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.212912 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.212919 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.212921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.213215 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.213475 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.213476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.213477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.213616 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.213920 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.213926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.214092 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.214220 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.214407 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":24,"thread_start_us":102,"threads_started":1}
I20260925 14:50:55.215093 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41533
I20260925 14:50:55.215196 7737 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.215413 7737 heartbeater.cc:511] Master 127.0.98.61:41533 requested a full tablet report, sending...
I20260925 14:50:55.215931 7096 ts_manager.cc:194] Registered new tserver with Master: 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:55.215994 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51547 (local address 127.0.98.60:32957)
0925 14:50:55.203670 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.203672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.203839 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:50:55.204251 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.204253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.204263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.204296 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.205053 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205552 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207458 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207773 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207843 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.210939 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.210940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.210940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.210948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.212606 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215025 (+ 2419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.215026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.215026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.215089 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215282 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.215283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.215283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.215449 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.215499 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.215790 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.215867 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:55.216171 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45317 (local address 127.0.98.62:45659)
0925 14:50:55.203305 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.203307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.203748 (+ 441us) server_negotiation.cc:407] Connection header received
0925 14:50:55.204248 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.204249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.204263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.204320 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.205073 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205828 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207017 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207216 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207312 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.212257 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.212259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.212259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.212266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.214482 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215231 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.215232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.215232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.215294 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215547 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.215547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.215548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.215675 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.215720 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.215992 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.216086 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:55.216346 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.98.3:51547)
0925 14:50:55.203776 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.203784 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.204116 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.204418 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.204423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.204774 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.204960 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.204964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205658 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.205658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.206517 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.206519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207842 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.207889 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.208258 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.208263 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.210225 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.213353 (+ 3128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.213354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.213355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.213590 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.215143 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.215144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.215202 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.215551 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.215551 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.215640 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.216119 (+ 479us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.216222 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":62,"threads_started":1}
I20260925 14:50:55.216578 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.203252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:45317)
0925 14:50:55.203688 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.203698 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.204105 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.204768 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.204769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.204915 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:55.204964 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.204965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.205921 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.205922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.206900 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.206902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.207319 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.207321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.207463 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.208408 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:55.208413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:55.211912 (+ 3499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:55.214587 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.214587 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.214588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.214814 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.215374 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:55.215374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:55.215375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:55.215472 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:55.215809 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:55.215810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:55.215848 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.216280 (+ 432us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.216393 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":92,"thread_start_us":96,"threads_started":1}
I20260925 14:50:55.217053 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.60:32957
I20260925 14:50:55.217057 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45659
I20260925 14:50:55.217113 7735 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.217114 7736 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:55.217186 7735 heartbeater.cc:511] Master 127.0.98.60:32957 requested a full tablet report, sending...
I20260925 14:50:55.217186 7736 heartbeater.cc:511] Master 127.0.98.62:45659 requested a full tablet report, sending...
I20260925 14:50:55.217410 6497 ts_manager.cc:194] Re-registered known tserver with Master: 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:55.217515 7282 ts_manager.cc:194] Registered new tserver with Master: 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:55.217878 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45317
I20260925 14:50:55.218245 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.208407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.0.98.61:41533)
0925 14:50:55.209204 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.209205 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.209333 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:50:55.209402 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.209403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.209417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.209477 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.210172 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.210955 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.212023 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.212343 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.212414 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.213913 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.213914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.213914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.213920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.215826 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.216188 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.216189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.216202 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.216255 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.216447 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.216448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.216448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.216566 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.217323 (+ 757us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.218064 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.218148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":59,"threads_started":1}
I20260925 14:50:55.222680 7333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.219659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.0.98.60:32957)
0925 14:50:55.219695 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.219697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.219706 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:55.219769 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.219770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.219793 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.219851 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.219995 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.220684 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.221980 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.222326 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.222459 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.222529 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.222595 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.222623 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:50:55.222689 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.219607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46912)
0925 14:50:55.219659 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.219669 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.219708 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.219888 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.219889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.219891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.219926 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.219927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.220775 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.220776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.221910 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.221911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.222467 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.222468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.222527 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.222531 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.222591 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.222630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:50:55.222991 7292 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 14:50:55.223186 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 517c35e223d647748517bc8379274670. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:55.223273 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:50:55.223446 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.208051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.0.98.62:45659)
0925 14:50:55.208859 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.208861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.209112 (+ 251us) server_negotiation.cc:407] Connection header received
0925 14:50:55.209160 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.209161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.209173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.209255 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.210071 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.210859 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.217669 (+ 6810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.217981 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.218044 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.220149 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.220150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.220150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.220162 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.221594 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.221909 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.221910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.221910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.221974 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.222191 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.222192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.222192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.222311 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.222420 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.222746 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.222852 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":88,"threads_started":1}
I20260925 14:50:55.224423 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.208740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.0.98.60:32957)
0925 14:50:55.208980 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.208982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.209525 (+ 543us) server_negotiation.cc:407] Connection header received
0925 14:50:55.209549 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.209550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.209562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.209619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.211356 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.213269 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.218121 (+ 4852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.218463 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.218544 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.219842 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.219843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.219843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.219851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.223028 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.223492 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.223494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.223494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.223551 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.223823 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.223824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.223824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.223951 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.224051 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.224223 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.224316 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":41,"threads_started":1}
I20260925 14:50:55.225208 6497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:55.226774 6497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:55.227487 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: true } } }
I20260925 14:50:55.227588 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:55.228685 7753 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: attempting to promote NON_VOTER 7292d19b964a481ab166a67930a02af4 to VOTER
I20260925 14:50:55.228834 7753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:55.229616 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 LEARNER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 14:50:55.229697 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Refusing update from remote peer 517c35e223d647748517bc8379274670: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 14:50:55.229779 7246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 14:50:55.229868 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 14:50:55.230633 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7292d19b964a481ab166a67930a02af4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.230808 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7292d19b964a481ab166a67930a02af4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.230847 7292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 7292d19b964a481ab166a67930a02af4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231544 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231626 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:55.231598 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231673 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:50:55.231686 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231769 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:55.231745 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231819 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:55.231834 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "517c35e223d647748517bc8379274670" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:55.231911 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:50:55.236884 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35201 (local address 127.0.0.1:45728)
0925 14:50:55.232995 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.233006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.233055 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.233575 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.233576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.233578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.233627 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.233629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.235006 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.235007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.235714 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.235716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236609 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.236700 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.236705 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.236754 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.236794 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":74,"threads_started":1}
I20260925 14:50:55.237128 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32957 (local address 127.0.0.1:42984)
0925 14:50:55.232776 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.232788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.232836 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.233341 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.233342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.233344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.233388 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.233389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.234973 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.234974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.236038 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.236039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236923 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.236991 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.236995 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.237043 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.237074 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1}
I20260925 14:50:55.237221 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.98.2:37901)
0925 14:50:55.233605 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.233610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.233620 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:55.233657 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.233659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.233677 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.233754 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.233822 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.234744 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236139 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236859 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236936 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.237003 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.237095 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.237127 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":69,"threads_started":1}
I20260925 14:50:55.237162 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.0.98.3:35201)
0925 14:50:55.233095 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.233101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.233112 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:50:55.233155 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.233158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.233175 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.233265 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.233719 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.234896 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.235818 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236425 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236494 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.236575 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.237047 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.237087 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260925 14:50:55.237445 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.98.1:32957)
0925 14:50:55.233135 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.233138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.233147 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:55.233180 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.233181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.233196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.233341 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.233441 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.234902 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236124 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236836 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.237239 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.237327 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.237351 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.237394 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260925 14:50:55.237797 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.232646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37901 (local address 127.0.0.1:52356)
0925 14:50:55.232900 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.232911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.232952 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.233746 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.233747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.233748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.233777 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.233778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.234845 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.234846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.236038 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.236040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.236936 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.236937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.237002 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.237006 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.237081 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.237120 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":49,"thread_start_us":77,"threads_started":1}
I20260925 14:50:55.237896 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 9858e603445d4d419a9202b435098c89 (DEFAULT_TABLE table=test-table [id=b38f3d4716db4d019a4abb74ab7a117a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:55.238312 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858e603445d4d419a9202b435098c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:55.238543 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 9858e603445d4d419a9202b435098c89 (DEFAULT_TABLE table=test-table [id=b38f3d4716db4d019a4abb74ab7a117a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:55.238867 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858e603445d4d419a9202b435098c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:55.238929 7397 tablet_service.cc:1467] Processing CreateTablet for tablet 9858e603445d4d419a9202b435098c89 (DEFAULT_TABLE table=test-table [id=b38f3d4716db4d019a4abb74ab7a117a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:55.239261 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9858e603445d4d419a9202b435098c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:55.242111 7767 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Bootstrap starting.
I20260925 14:50:55.242194 7765 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Bootstrap starting.
I20260925 14:50:55.242501 7766 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Bootstrap starting.
I20260925 14:50:55.243109 7766 tablet_bootstrap.cc:654] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:55.243103 7765 tablet_bootstrap.cc:654] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:55.243103 7767 tablet_bootstrap.cc:654] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:55.243407 7766 log.cc:824] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:55.243453 7765 log.cc:824] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:55.243453 7767 log.cc:824] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:55.244326 7767 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: No bootstrap required, opened a new log
I20260925 14:50:55.244410 7767 ts_tablet_manager.cc:1397] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:55.244577 7766 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: No bootstrap required, opened a new log
I20260925 14:50:55.244601 7765 tablet_bootstrap.cc:492] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: No bootstrap required, opened a new log
I20260925 14:50:55.244650 7766 ts_tablet_manager.cc:1397] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:55.244650 7765 ts_tablet_manager.cc:1397] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:50:55.246335 7767 raft_consensus.cc:348] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.246562 7767 raft_consensus.cc:374] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:55.246649 7765 raft_consensus.cc:348] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.246832 7765 raft_consensus.cc:374] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:55.246874 7765 raft_consensus.cc:729] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c862c5525d4478496446ad8ecf53d52, State: Initialized, Role: FOLLOWER
I20260925 14:50:55.246745 7767 raft_consensus.cc:729] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9857b70164976ae22a6faf92b58cc, State: Initialized, Role: FOLLOWER
I20260925 14:50:55.247007 7765 consensus_queue.cc:260] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.247047 7767 consensus_queue.cc:260] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.247157 7766 raft_consensus.cc:348] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.247267 7765 ts_tablet_manager.cc:1428] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:55.247308 7766 raft_consensus.cc:374] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:55.247341 7766 raft_consensus.cc:729] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394d329381b2487283d737bbcc863193, State: Initialized, Role: FOLLOWER
I20260925 14:50:55.247356 7767 ts_tablet_manager.cc:1428] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:55.247467 7600 heartbeater.cc:503] Master 127.0.98.62:45659 was elected leader, sending a full tablet report...
I20260925 14:50:55.247473 7766 consensus_queue.cc:260] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.247684 7766 ts_tablet_manager.cc:1428] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:50:55.247984 7736 heartbeater.cc:503] Master 127.0.98.62:45659 was elected leader, sending a full tablet report...
I20260925 14:50:55.248932 7464 heartbeater.cc:503] Master 127.0.98.62:45659 was elected leader, sending a full tablet report...
W20260925 14:50:55.326627 7602 tablet.cc:2369] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:55.435568 7466 tablet.cc:2369] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:50:55.453347 7738 tablet.cc:2369] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:50:55.573621 7771 raft_consensus.cc:482] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:55.573800 7771 raft_consensus.cc:504] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.574357 7771 leader_election.cc:288] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901), 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:55.577448 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.574664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50925 (local address 127.0.98.2:37901)
0925 14:50:55.574714 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.574716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.574729 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:50:55.574817 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.574818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.574832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.574879 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.575030 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.575591 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.576722 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.577099 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.577152 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.577161 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.577327 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.577370 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:50:55.577436 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.574530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37901 (local address 127.0.98.1:50925)
0925 14:50:55.574665 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.574676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.574747 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.574893 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.574894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.574896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.574955 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.574957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.575661 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.575662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.576623 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.576625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.577166 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.577167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.577235 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.577239 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.577305 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.577350 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":19,"thread_start_us":62,"threads_started":1}
I20260925 14:50:55.577615 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.574641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38133 (local address 127.0.98.3:35201)
0925 14:50:55.574719 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.574722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.574758 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:50:55.574848 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.574848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.574859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.574917 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:55.575097 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.575704 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.576686 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.577252 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.577314 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.577323 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.577497 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.577540 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:50:55.577775 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.574547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35201 (local address 127.0.98.1:38133)
0925 14:50:55.574697 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:55.574707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:55.574769 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:55.574933 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:55.574934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:55.574935 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:55.575022 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.575023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.575793 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.575794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.576608 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:55.576610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.577332 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.577333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:55.577406 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.577410 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:55.577467 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:50:55.577650 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260925 14:50:55.577772 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858e603445d4d419a9202b435098c89" candidate_uuid: "4c862c5525d4478496446ad8ecf53d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9857b70164976ae22a6faf92b58cc" is_pre_election: true
I20260925 14:50:55.577948 7553 raft_consensus.cc:2454] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c862c5525d4478496446ad8ecf53d52 in term 0.
I20260925 14:50:55.578184 7352 leader_election.cc:302] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9857b70164976ae22a6faf92b58cc, 4c862c5525d4478496446ad8ecf53d52; no voters:
I20260925 14:50:55.578207 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858e603445d4d419a9202b435098c89" candidate_uuid: "4c862c5525d4478496446ad8ecf53d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394d329381b2487283d737bbcc863193" is_pre_election: true
I20260925 14:50:55.578393 7689 raft_consensus.cc:2454] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c862c5525d4478496446ad8ecf53d52 in term 0.
I20260925 14:50:55.578404 7771 raft_consensus.cc:2781] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:55.578450 7771 raft_consensus.cc:482] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:55.578485 7771 raft_consensus.cc:3037] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:55.579159 7771 raft_consensus.cc:504] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.579306 7771 leader_election.cc:288] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [CANDIDATE]: Term 1 election: Requested vote from peers 13d9857b70164976ae22a6faf92b58cc (127.0.98.2:37901), 394d329381b2487283d737bbcc863193 (127.0.98.3:35201)
I20260925 14:50:55.579447 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858e603445d4d419a9202b435098c89" candidate_uuid: "4c862c5525d4478496446ad8ecf53d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394d329381b2487283d737bbcc863193"
I20260925 14:50:55.579443 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9858e603445d4d419a9202b435098c89" candidate_uuid: "4c862c5525d4478496446ad8ecf53d52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13d9857b70164976ae22a6faf92b58cc"
I20260925 14:50:55.579522 7553 raft_consensus.cc:3037] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:55.579523 7689 raft_consensus.cc:3037] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:55.580382 7689 raft_consensus.cc:2454] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c862c5525d4478496446ad8ecf53d52 in term 1.
I20260925 14:50:55.580382 7553 raft_consensus.cc:2454] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c862c5525d4478496446ad8ecf53d52 in term 1.
I20260925 14:50:55.580549 7350 leader_election.cc:302] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394d329381b2487283d737bbcc863193, 4c862c5525d4478496446ad8ecf53d52; no voters:
I20260925 14:50:55.580648 7771 raft_consensus.cc:2781] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:55.580756 7771 raft_consensus.cc:686] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 1 LEADER]: Becoming Leader. State: Replica: 4c862c5525d4478496446ad8ecf53d52, State: Running, Role: LEADER
I20260925 14:50:55.580870 7771 consensus_queue.cc:237] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } }
I20260925 14:50:55.581619 6497 catalog_manager.cc:5495] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4c862c5525d4478496446ad8ecf53d52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:55.674413 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.671873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.0.98.1:32957)
0925 14:50:55.671936 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.671938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.671948 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:55.671994 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.671995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.672014 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.672096 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:55.672244 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.672720 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.673544 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.673757 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.673851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.674017 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:55.674121 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:55.674253 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.674276 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.674298 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:50:55.676378 7553 raft_consensus.cc:1261] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Refusing update from remote peer 4c862c5525d4478496446ad8ecf53d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:55.676386 7689 raft_consensus.cc:1261] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Refusing update from remote peer 4c862c5525d4478496446ad8ecf53d52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:55.676771 7776 consensus_queue.cc:1035] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:55.676954 7771 consensus_queue.cc:1035] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:55.679486 7780 mvcc.cc:205] Tried to move back new op lower bound from 7333264816847155200 to 7333264816459784192. Current Snapshot: MvccSnapshot[applied={T|T < 7333264816847155200}]
I20260925 14:50:55.683272 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:55.683369 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:55.684497 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 517c35e223d647748517bc8379274670: Sending DeleteTablet for 3 replicas of tablet 9858e603445d4d419a9202b435098c89
I20260925 14:50:55.684859 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858e603445d4d419a9202b435098c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42984
I20260925 14:50:55.685056 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858e603445d4d419a9202b435098c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45728
I20260925 14:50:55.685245 7789 tablet_replica.cc:330] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: stopping tablet replica
I20260925 14:50:55.685355 7789 raft_consensus.cc:2229] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:55.685465 7789 raft_consensus.cc:2258] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:55.685986 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 9858e603445d4d419a9202b435098c89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:55 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52356
I20260925 14:50:55.686309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6466
I20260925 14:50:55.690764 7789 ts_tablet_manager.cc:1905] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:55.692700 7789 ts_tablet_manager.cc:1918] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:55.692791 7789 log.cc:1196] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/9858e603445d4d419a9202b435098c89
I20260925 14:50:55.693056 7789 ts_tablet_manager.cc:1939] T 9858e603445d4d419a9202b435098c89 P 394d329381b2487283d737bbcc863193: Deleting consensus metadata
W20260925 14:50:55.693434 7625 connection.cc:441] server connection from 127.0.0.1:45728 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45728 (request call id 1) could send its response
I20260925 14:50:55.693465 7788 tablet_replica.cc:330] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: stopping tablet replica
I20260925 14:50:55.693578 7788 raft_consensus.cc:2229] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:50:55.693737 7788 raft_consensus.cc:2258] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:55.693893 7790 tablet_replica.cc:330] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: stopping tablet replica
I20260925 14:50:55.693994 7790 raft_consensus.cc:2229] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:55.694121 7790 raft_consensus.cc:2258] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:55.694571 7788 ts_tablet_manager.cc:1905] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:55.696156 7788 ts_tablet_manager.cc:1918] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:55.696236 7788 log.cc:1196] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/9858e603445d4d419a9202b435098c89
I20260925 14:50:55.696486 7788 ts_tablet_manager.cc:1939] T 9858e603445d4d419a9202b435098c89 P 4c862c5525d4478496446ad8ecf53d52: Deleting consensus metadata
I20260925 14:50:55.696923 7790 ts_tablet_manager.cc:1905] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:50:55.696992 7352 connection.cc:441] server connection from 127.0.0.1:42984 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42984 (request call id 1) could send its response
I20260925 14:50:55.698453 7790 ts_tablet_manager.cc:1918] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:55.698530 7790 log.cc:1196] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/9858e603445d4d419a9202b435098c89
I20260925 14:50:55.698820 7790 ts_tablet_manager.cc:1939] T 9858e603445d4d419a9202b435098c89 P 13d9857b70164976ae22a6faf92b58cc: Deleting consensus metadata
W20260925 14:50:55.699216 7489 connection.cc:441] server connection from 127.0.0.1:52356 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52356 (request call id 1) could send its response
I20260925 14:50:55.702013 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.694374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.61:41533)
0925 14:50:55.694419 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.694421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.694528 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:50:55.694624 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.694625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.694639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.694669 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.695459 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.696244 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.697300 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.697618 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.697690 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.699410 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.699411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.699411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.699419 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.700949 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.701296 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.701297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.701298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.701358 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.701535 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.701536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.701537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.701664 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.701762 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.701837 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.701919 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:55.708441 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:55.694565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.0.98.60:32957)
0925 14:50:55.694607 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:55.694609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:55.702036 (+ 7427us) server_negotiation.cc:407] Connection header received
0925 14:50:55.702147 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:55.702148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:55.702164 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:55.702191 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:55.702766 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.703527 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.704222 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:55.704543 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:55.704614 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:55.705660 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:55.705661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:55.705661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:55.705669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:55.707271 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.707595 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.707597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.707597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.707679 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:55.707879 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:55.707880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:55.707881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:55.708006 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:55.708102 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:55.708280 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:50:55.708347 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:50:55.786798 7313 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.60:32957
W20260925 14:50:55.935658 7458 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.1:32957
I20260925 14:50:56.681128 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:56.680838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:40823)
0925 14:50:56.680992 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:56.681060 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
W20260925 14:50:56.681486 7736 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:56.681639 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:56.681268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:54539)
0925 14:50:56.681473 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:56.681547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":52,"threads_started":1}
W20260925 14:50:56.681815 7464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:56.682385 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:56.682098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:38031)
0925 14:50:56.682254 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:56.682318 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
W20260925 14:50:56.682593 7600 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.186036 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 517c35e223d647748517bc8379274670)
I20260925 14:50:57.186151 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 517c35e223d647748517bc8379274670)
I20260925 14:50:57.186192 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:57.186245 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:57.186457 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 7292d19b964a481ab166a67930a02af4 (127.0.98.60:32957)
W20260925 14:50:57.186717 7317 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.98.60:32957: Invalid argument: socket address is ambiguous
I20260925 14:50:57.186755 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61:41533)
I20260925 14:50:57.186936 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46904)
0925 14:50:57.186851 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.186899 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":39,"threads_started":1}
W20260925 14:50:57.187011 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.187114 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46892)
0925 14:50:57.186896 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.186962 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":64,"threads_started":1}
W20260925 14:50:57.187317 7082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.187398 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.187316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46916)
0925 14:50:57.187350 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.187372 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 14:50:57.187479 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.187865 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.187743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46922)
0925 14:50:57.187810 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.187840 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:50:57.187987 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.189941 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41533 (local address 127.0.0.1:48420)
0925 14:50:57.186860 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.186871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:57.186926 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:57.187232 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:57.187233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:57.187236 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:57.187291 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:57.187293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.188243 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.188244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:57.189066 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:57.189068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.189687 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:57.189759 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:57.189762 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:57.189832 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:50:57.189874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":43,"threads_started":1}
I20260925 14:50:57.189940 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.98.61:41533)
0925 14:50:57.187029 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:57.187031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:57.187038 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:50:57.187061 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:57.187061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:57.187073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:57.187147 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:57.187368 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.188137 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189173 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.189602 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189677 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:57.189689 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:57.189847 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:50:57.189883 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":85,"thread_start_us":36,"threads_started":1}
I20260925 14:50:57.190155 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.60:32957)
0925 14:50:57.187006 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:57.187008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:57.187186 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:50:57.187210 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:57.187211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:57.187221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:57.187306 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:57.187487 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.188207 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189290 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.189657 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189746 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:57.189756 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:57.189889 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:50:57.189927 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":51,"threads_started":1}
I20260925 14:50:57.190171 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.186605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32957 (local address 127.0.0.1:46928)
0925 14:50:57.187060 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.187069 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:57.187129 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:57.187353 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:57.187354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:57.187356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:57.187408 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:57.187410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.188311 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.188312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:57.189209 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:57.189211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:57.189750 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:57.189751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:57.189823 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:57.189827 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:57.189883 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:50:57.189926 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":47,"threads_started":1}
I20260925 14:50:57.190157 7106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7292d19b964a481ab166a67930a02af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53f0eae7a77749c3adee8749ff80ec2a" is_pre_election: true
I20260925 14:50:57.190470 7106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7292d19b964a481ab166a67930a02af4 in term 1.
I20260925 14:50:57.190516 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53f0eae7a77749c3adee8749ff80ec2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7292d19b964a481ab166a67930a02af4" is_pre_election: true
I20260925 14:50:57.190621 7292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f0eae7a77749c3adee8749ff80ec2a in term 1.
I20260925 14:50:57.190649 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a, 7292d19b964a481ab166a67930a02af4; no voters: 517c35e223d647748517bc8379274670
I20260925 14:50:57.190749 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:50:57.190790 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 517c35e223d647748517bc8379274670)
I20260925 14:50:57.190771 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a, 7292d19b964a481ab166a67930a02af4; no voters: 517c35e223d647748517bc8379274670
I20260925 14:50:57.190821 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:50:57.190894 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:50:57.190928 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 517c35e223d647748517bc8379274670)
I20260925 14:50:57.190953 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:50:57.191583 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:57.191596 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:57.191740 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 election: Requested vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 7292d19b964a481ab166a67930a02af4 (127.0.98.60:32957)
I20260925 14:50:57.191840 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 election: Requested vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61:41533)
I20260925 14:50:57.191923 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.191788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46932)
0925 14:50:57.191879 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.191900 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 14:50:57.191902 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53f0eae7a77749c3adee8749ff80ec2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7292d19b964a481ab166a67930a02af4"
I20260925 14:50:57.191984 7292 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 53f0eae7a77749c3adee8749ff80ec2a in current term 2: Already voted for candidate 7292d19b964a481ab166a67930a02af4 in this term.
I20260925 14:50:57.192008 7106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7292d19b964a481ab166a67930a02af4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53f0eae7a77749c3adee8749ff80ec2a"
I20260925 14:50:57.192080 7106 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7292d19b964a481ab166a67930a02af4 in current term 2: Already voted for candidate 53f0eae7a77749c3adee8749ff80ec2a in this term.
I20260925 14:50:57.192157 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.191784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46924)
0925 14:50:57.191828 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.191855 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:50:57.192287 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.192209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46948)
0925 14:50:57.192237 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.192265 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 14:50:57.192364 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.192420 7082 leader_election.cc:302] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a; no voters: 517c35e223d647748517bc8379274670, 7292d19b964a481ab166a67930a02af4
I20260925 14:50:57.192478 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 14:50:57.192529 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.192451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:46956)
0925 14:50:57.192488 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.192509 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:50:57.192596 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:57.192654 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7292d19b964a481ab166a67930a02af4; no voters: 517c35e223d647748517bc8379274670, 53f0eae7a77749c3adee8749ff80ec2a
I20260925 14:50:57.192737 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 14:50:57.682145 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.681887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:53815)
0925 14:50:57.682043 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.682083 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260925 14:50:57.682379 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.682082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:35189)
0925 14:50:57.682237 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.682286 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":48,"threads_started":1}
I20260925 14:50:57.683053 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:57.682862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:48955)
0925 14:50:57.682957 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:57.683004 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260925 14:50:58.682828 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:58.682529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:32887)
0925 14:50:58.682700 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:58.682769 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260925 14:50:58.683007 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:58.682817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:46413)
0925 14:50:58.682923 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:58.682959 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260925 14:50:58.683640 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:58.683437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:39695)
0925 14:50:58.683578 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:58.683608 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":56,"threads_started":1}
I20260925 14:50:59.002591 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:59.002703 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:59.002880 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61:41533)
I20260925 14:50:59.003171 7106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7292d19b964a481ab166a67930a02af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53f0eae7a77749c3adee8749ff80ec2a" is_pre_election: true
I20260925 14:50:59.003222 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.003017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47214)
0925 14:50:59.003126 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.003180 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260925 14:50:59.003285 7106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7292d19b964a481ab166a67930a02af4 in term 2.
I20260925 14:50:59.003515 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a, 7292d19b964a481ab166a67930a02af4; no voters:
I20260925 14:50:59.003634 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 14:50:59.003715 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.003569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47224)
0925 14:50:59.003640 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.003668 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 14:50:59.003777 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:59.003715 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:59.003873 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:50:59.004515 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:59.004640 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 election: Requested vote from peers 517c35e223d647748517bc8379274670 (127.0.98.62:45659), 53f0eae7a77749c3adee8749ff80ec2a (127.0.98.61:41533)
I20260925 14:50:59.004854 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.004736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47226)
0925 14:50:59.004794 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.004827 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:50:59.004827 7106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7292d19b964a481ab166a67930a02af4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "53f0eae7a77749c3adee8749ff80ec2a"
I20260925 14:50:59.004904 7106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:50:59.005224 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.005128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47232)
0925 14:50:59.005177 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.005204 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:50:59.005311 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
I20260925 14:50:59.005685 7106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7292d19b964a481ab166a67930a02af4 in term 3.
I20260925 14:50:59.005861 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53f0eae7a77749c3adee8749ff80ec2a, 7292d19b964a481ab166a67930a02af4; no voters: 517c35e223d647748517bc8379274670
I20260925 14:50:59.005959 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:50:59.006099 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [term 3 LEADER]: Becoming Leader. State: Replica: 7292d19b964a481ab166a67930a02af4, State: Running, Role: LEADER
I20260925 14:50:59.006199 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } }
I20260925 14:50:59.006686 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7292d19b964a481ab166a67930a02af4. Latest consensus state: current_term: 3 leader_uuid: "7292d19b964a481ab166a67930a02af4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:59.006798 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:59.006960 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:59.007328 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:50:59.007385 7821 catalog_manager.cc:1259] Loaded cluster ID: 30232186eaf64c3a811ff2153377b04d
I20260925 14:50:59.007401 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:59.007541 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:59.007654 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Loaded TSK: 0
I20260925 14:50:59.007836 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:50:59.030448 7282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:59.032021 7282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:59.033068 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.032944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47248)
0925 14:50:59.033002 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.033037 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:50:59.033052 7106 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [term 3 FOLLOWER]: Refusing update from remote peer 7292d19b964a481ab166a67930a02af4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 14:50:59.033265 7819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 14:50:59.033414 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.033314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.0.1:47254)
0925 14:50:59.033364 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.033393 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:50:59.033481 7268 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 -> Peer 517c35e223d647748517bc8379274670 (127.0.98.62:45659): Couldn't send request to peer 517c35e223d647748517bc8379274670. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:50:59.033893 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7292d19b964a481ab166a67930a02af4. Latest consensus state: current_term: 3 leader_uuid: "7292d19b964a481ab166a67930a02af4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:59.034017 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:50:59.034348 7822 mvcc.cc:205] Tried to move back new op lower bound from 7333264830597378048 to 7333264830490349568. Current Snapshot: MvccSnapshot[applied={T|T < 7333264830597378048}]
I20260925 14:50:59.034472 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "7292d19b964a481ab166a67930a02af4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:59.034567 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:59.034680 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "7292d19b964a481ab166a67930a02af4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "517c35e223d647748517bc8379274670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } peers { permanent_uuid: "53f0eae7a77749c3adee8749ff80ec2a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41533 } attrs { promote: false } } peers { permanent_uuid: "7292d19b964a481ab166a67930a02af4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32957 } attrs { promote: false } } }
I20260925 14:50:59.034771 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53f0eae7a77749c3adee8749ff80ec2a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:50:59.039966 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.0.98.1:32957)
0925 14:50:59.037081 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.037082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.037090 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:50:59.037115 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.037115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.037125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.037188 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:59.037343 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.038049 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.039146 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.039615 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.039693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.039706 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.039854 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.039895 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
I20260925 14:50:59.039973 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32957 (local address 127.0.0.1:43002)
0925 14:50:59.036932 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.036943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:59.036987 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:59.037235 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:59.037236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:59.037238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:59.037270 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.037271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.038132 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.038132 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.039077 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.039079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.039704 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.039705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.039776 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.039781 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:50:59.039842 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:50:59.039883 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20260925 14:50:59.040253 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37901 (local address 127.0.0.1:52372)
0925 14:50:59.036811 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.036821 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:59.036868 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:59.037207 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:59.037207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:59.037209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:59.037255 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.037256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.037921 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.037923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.039228 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.039231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.040064 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.040065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.040133 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.040136 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:59.040167 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 14:50:59.040207 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260925 14:50:59.040365 7397 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b9f31353e4619ba0ad0360119affe (DEFAULT_TABLE table=test-table [id=a84ece0a822945978066243eeada137e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:59.040499 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b9f31353e4619ba0ad0360119affe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:59.040542 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.0.98.2:37901)
0925 14:50:59.037085 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.037087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.037096 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:59.037124 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.037125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.037136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.037206 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:59.037313 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.037835 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.039349 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.039958 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.040401 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.040419 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.040441 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.040482 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":58,"threads_started":1}
I20260925 14:50:59.040695 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35201 (local address 127.0.0.1:45732)
0925 14:50:59.036753 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.036767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:59.036826 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:59.037063 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:59.037064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:59.037066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:59.037120 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.037122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.038120 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.038121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.039382 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.039383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.040455 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.040456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.040520 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.040524 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:59.040570 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:50:59.040616 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":19}
I20260925 14:50:59.040777 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b9f31353e4619ba0ad0360119affe (DEFAULT_TABLE table=test-table [id=a84ece0a822945978066243eeada137e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:59.040879 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.036691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.0.98.3:35201)
0925 14:50:59.036813 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.036816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.036828 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:50:59.036941 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.036941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.036953 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.037012 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:59.037245 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.038036 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.039461 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.040344 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.040411 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.040423 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.040771 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.040813 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260925 14:50:59.040903 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b9f31353e4619ba0ad0360119affe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:59.041108 7669 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0b9f31353e4619ba0ad0360119affe (DEFAULT_TABLE table=test-table [id=a84ece0a822945978066243eeada137e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:59.041220 7669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0b9f31353e4619ba0ad0360119affe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:59.042439 7833 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52: Bootstrap starting.
I20260925 14:50:59.042447 7834 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: Bootstrap starting.
I20260925 14:50:59.042763 7835 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Bootstrap starting.
I20260925 14:50:59.042805 7834 tablet_bootstrap.cc:654] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:59.042806 7833 tablet_bootstrap.cc:654] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:59.043226 7835 tablet_bootstrap.cc:654] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:59.043382 7833 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52: No bootstrap required, opened a new log
I20260925 14:50:59.043447 7833 ts_tablet_manager.cc:1397] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:59.043558 7833 raft_consensus.cc:348] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.043639 7833 raft_consensus.cc:374] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:59.043676 7833 raft_consensus.cc:729] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c862c5525d4478496446ad8ecf53d52, State: Initialized, Role: FOLLOWER
I20260925 14:50:59.043713 7833 consensus_queue.cc:260] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.043864 7835 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: No bootstrap required, opened a new log
I20260925 14:50:59.043877 7833 ts_tablet_manager.cc:1428] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:50:59.043912 7835 ts_tablet_manager.cc:1397] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:59.044072 7835 raft_consensus.cc:348] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.044200 7835 raft_consensus.cc:374] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:59.044246 7835 raft_consensus.cc:729] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394d329381b2487283d737bbcc863193, State: Initialized, Role: FOLLOWER
I20260925 14:50:59.044328 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.044151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:51115)
0925 14:50:59.044265 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.044297 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 14:50:59.044329 7835 consensus_queue.cc:260] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.044482 7834 tablet_bootstrap.cc:492] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: No bootstrap required, opened a new log
I20260925 14:50:59.044530 7834 ts_tablet_manager.cc:1397] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:50:59.044528 7835 ts_tablet_manager.cc:1428] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:59.044767 7834 raft_consensus.cc:348] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.044847 7834 raft_consensus.cc:374] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:59.044868 7834 raft_consensus.cc:729] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d9857b70164976ae22a6faf92b58cc, State: Initialized, Role: FOLLOWER
I20260925 14:50:59.044929 7834 consensus_queue.cc:260] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.045020 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.044921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:52075)
0925 14:50:59.044959 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.044993 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:50:59.045079 7735 heartbeater.cc:503] Master 127.0.98.60:32957 was elected leader, sending a full tablet report...
I20260925 14:50:59.045096 7834 ts_tablet_manager.cc:1428] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:59.045488 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.045325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:33173)
0925 14:50:59.045427 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.045461 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 14:50:59.045935 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.045791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:56001)
0925 14:50:59.045864 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.045898 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:50:59.046187 7599 heartbeater.cc:503] Master 127.0.98.60:32957 was elected leader, sending a full tablet report...
I20260925 14:50:59.046479 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.046394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:34029)
0925 14:50:59.046432 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.046457 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:50:59.090091 7841 raft_consensus.cc:482] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:59.090273 7841 raft_consensus.cc:504] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.090649 7841 leader_election.cc:288] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 394d329381b2487283d737bbcc863193 (127.0.98.3:35201), 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
I20260925 14:50:59.094795 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.090910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43229 (local address 127.0.98.3:35201)
0925 14:50:59.090980 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.090983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.090992 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:50:59.091021 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.091022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.091034 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.091097 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:59.091450 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.092226 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.093818 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.094460 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.094526 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.094539 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.094683 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.094726 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:50:59.094808 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.091053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54777 (local address 127.0.98.1:32957)
0925 14:50:59.091100 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.091103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.091181 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:50:59.091235 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.091236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.091246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.091296 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:59.091532 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.092230 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.093810 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.094431 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.094515 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.094529 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.094682 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.094728 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:50:59.094820 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.090830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35201 (local address 127.0.98.2:43229)
0925 14:50:59.090888 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.090901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:59.090952 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:59.091124 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:59.091125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:59.091127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:59.091257 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.091259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.092334 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.092335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.093677 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.093679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.094529 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.094530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.094596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.094604 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:50:59.094662 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:50:59.094737 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":83}
I20260925 14:50:59.094820 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.090948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32957 (local address 127.0.98.2:54777)
0925 14:50:59.091126 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.091136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:59.091188 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:59.091403 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:59.091404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:59.091407 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:59.091457 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.091459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.092331 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.092333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.093677 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:59.093679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.094527 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.094529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:59.094596 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.094602 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:50:59.094662 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:50:59.094737 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":99,"thread_start_us":106,"threads_started":1}
I20260925 14:50:59.095042 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b9f31353e4619ba0ad0360119affe" candidate_uuid: "13d9857b70164976ae22a6faf92b58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394d329381b2487283d737bbcc863193" is_pre_election: true
I20260925 14:50:59.095147 7689 raft_consensus.cc:2454] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d9857b70164976ae22a6faf92b58cc in term 0.
I20260925 14:50:59.095340 7486 leader_election.cc:302] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9857b70164976ae22a6faf92b58cc, 394d329381b2487283d737bbcc863193; no voters:
I20260925 14:50:59.095331 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b9f31353e4619ba0ad0360119affe" candidate_uuid: "13d9857b70164976ae22a6faf92b58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c862c5525d4478496446ad8ecf53d52" is_pre_election: true
I20260925 14:50:59.095471 7417 raft_consensus.cc:2454] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d9857b70164976ae22a6faf92b58cc in term 0.
I20260925 14:50:59.095484 7841 raft_consensus.cc:2781] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:59.095547 7841 raft_consensus.cc:482] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:59.095574 7841 raft_consensus.cc:3037] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:59.096439 7841 raft_consensus.cc:504] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.096580 7841 leader_election.cc:288] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [CANDIDATE]: Term 1 election: Requested vote from peers 394d329381b2487283d737bbcc863193 (127.0.98.3:35201), 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957)
I20260925 14:50:59.096787 7689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b9f31353e4619ba0ad0360119affe" candidate_uuid: "13d9857b70164976ae22a6faf92b58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "394d329381b2487283d737bbcc863193"
I20260925 14:50:59.096773 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0b9f31353e4619ba0ad0360119affe" candidate_uuid: "13d9857b70164976ae22a6faf92b58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c862c5525d4478496446ad8ecf53d52"
I20260925 14:50:59.096864 7689 raft_consensus.cc:3037] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:59.096868 7417 raft_consensus.cc:3037] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:59.097402 7689 raft_consensus.cc:2454] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d9857b70164976ae22a6faf92b58cc in term 1.
I20260925 14:50:59.097580 7486 leader_election.cc:302] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d9857b70164976ae22a6faf92b58cc, 394d329381b2487283d737bbcc863193; no voters:
I20260925 14:50:59.097604 7417 raft_consensus.cc:2454] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d9857b70164976ae22a6faf92b58cc in term 1.
I20260925 14:50:59.097703 7841 raft_consensus.cc:2781] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:59.097915 7841 raft_consensus.cc:686] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 1 LEADER]: Becoming Leader. State: Replica: 13d9857b70164976ae22a6faf92b58cc, State: Running, Role: LEADER
I20260925 14:50:59.098009 7841 consensus_queue.cc:237] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } }
I20260925 14:50:59.098409 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.098194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:55465)
0925 14:50:59.098326 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.098381 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 14:50:59.098799 7281 catalog_manager.cc:5495] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d9857b70164976ae22a6faf92b58cc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "13d9857b70164976ae22a6faf92b58cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d9857b70164976ae22a6faf92b58cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37901 } health_report { overall_health: HEALTHY } } }
I20260925 14:50:59.114313 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.112347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.0.98.2:37901)
0925 14:50:59.112415 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:59.112416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:59.112426 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:59.112464 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:59.112465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:59.112480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:59.112539 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:59.112643 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.113085 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.113729 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:59.113906 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:59.113952 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:59.114072 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:59.114161 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:59.114211 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:59.114248 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:50:59.114266 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:50:59.115419 7689 raft_consensus.cc:1261] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Refusing update from remote peer 13d9857b70164976ae22a6faf92b58cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:59.115499 7417 raft_consensus.cc:1261] T 1c0b9f31353e4619ba0ad0360119affe P 4c862c5525d4478496446ad8ecf53d52 [term 1 FOLLOWER]: Refusing update from remote peer 13d9857b70164976ae22a6faf92b58cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:59.115688 7841 consensus_queue.cc:1035] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "394d329381b2487283d737bbcc863193" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:59.115829 7843 consensus_queue.cc:1035] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c862c5525d4478496446ad8ecf53d52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:59.116581 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.116411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:57501)
0925 14:50:59.116507 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.116549 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 14:50:59.116652 7463 heartbeater.cc:503] Master 127.0.98.60:32957 was elected leader, sending a full tablet report...
I20260925 14:50:59.117117 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.116971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:40541)
0925 14:50:59.117059 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.117086 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:50:59.117967 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.117884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.3:48841)
0925 14:50:59.117919 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.117942 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:50:59.118831 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.117742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:49959)
0925 14:50:59.118772 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.118805 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1012}
I20260925 14:50:59.119041 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.118909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.1:56383)
0925 14:50:59.118989 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.119018 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 14:50:59.119230 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.119110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45659 (local address 127.0.98.2:43949)
0925 14:50:59.119174 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.119201 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 14:50:59.119799 7850 mvcc.cc:205] Tried to move back new op lower bound from 7333264830934269952 to 7333264830865956864. Current Snapshot: MvccSnapshot[applied={T|T < 7333264830934269952}]
I20260925 14:50:59.119886 7282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:59.119971 7282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:59.121206 7282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7292d19b964a481ab166a67930a02af4: Sending DeleteTablet for 3 replicas of tablet 1c0b9f31353e4619ba0ad0360119affe
I20260925 14:50:59.121462 7533 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b9f31353e4619ba0ad0360119affe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:52372
I20260925 14:50:59.121631 7669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b9f31353e4619ba0ad0360119affe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45732
I20260925 14:50:59.121814 7858 tablet_replica.cc:330] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: stopping tablet replica
I20260925 14:50:59.121876 7858 raft_consensus.cc:2229] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:59.121887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7334
I20260925 14:50:59.121915 7858 raft_consensus.cc:2258] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:59.121905 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c0b9f31353e4619ba0ad0360119affe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:59 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43002
I20260925 14:50:59.123433 7858 ts_tablet_manager.cc:1905] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:59.125000 7858 ts_tablet_manager.cc:1918] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:59.125066 7858 log.cc:1196] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/1c0b9f31353e4619ba0ad0360119affe
I20260925 14:50:59.125325 7858 ts_tablet_manager.cc:1939] T 1c0b9f31353e4619ba0ad0360119affe P 394d329381b2487283d737bbcc863193: Deleting consensus metadata
I20260925 14:50:59.125691 7266 catalog_manager.cc:4841] TS 394d329381b2487283d737bbcc863193 (127.0.98.3:35201): tablet 1c0b9f31353e4619ba0ad0360119affe (table test-table [id=a84ece0a822945978066243eeada137e]) successfully deleted
I20260925 14:50:59.128371 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.128232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32957 (local address 127.0.0.1:43014)
0925 14:50:59.128292 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.128322 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:50:59.128449 7857 tablet_replica.cc:330] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: stopping tablet replica
I20260925 14:50:59.128494 7857 raft_consensus.cc:2229] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 1 LEADER]: Raft consensus shutting down.
W20260925 14:50:59.128509 7266 catalog_manager.cc:4584] TS 4c862c5525d4478496446ad8ecf53d52 (127.0.98.1:32957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c0b9f31353e4619ba0ad0360119affe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32957: connect: Connection refused (error 111)
I20260925 14:50:59.128566 7857 raft_consensus.cc:2258] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:59.128783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7470
I20260925 14:50:59.128801 7857 ts_tablet_manager.cc:1905] T 1c0b9f31353e4619ba0ad0360119affe P 13d9857b70164976ae22a6faf92b58cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:59.134480 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:59.134366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37901 (local address 127.0.0.1:52382)
0925 14:50:59.134414 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:59.134443 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:50:59.134626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7606
I20260925 14:50:59.138829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 6962
I20260925 14:50:59.143754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7150
2026-09-25T14:50:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6880 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:59 dist-test-slave-gpc1 krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:59 dist-test-slave-gpc1 krb5kdc[7861](info): set up 2 sockets
Sep 25 14:50:59 dist-test-slave-gpc1 krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260925 14:51:00.170862 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 14:51:00.288841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:00.367853 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:00.368079 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:00.368116 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:00.369359 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:00.369423 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:51:00.369444 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:00.369468 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:00.369496 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:00.369525 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:00.370987 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43645
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:00.371268 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:00.371492 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:00.376323 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.376294 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.376384 7874 server_base.cc:1034] running on GCE node
W20260925 14:51:00.376304 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.376796 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:00.377125 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:00.378304 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790347860378279 us; error 37 us; skew 500 ppm
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:00.380808 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 14:51:00.381186 7874 webserver.cc:460] Webserver started at http://127.0.98.62:41731/ using document root <none> and password file <none>
I20260925 14:51:00.381374 7874 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:00.381413 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:00.381496 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:00.382611 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "25fd120315a848a88d8be6c7ee44705e"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cad9c5b362e5bbe04167bb926480419e"
server_key_iv: "333616604413f0bda41bae73d692210b"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.382979 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "25fd120315a848a88d8be6c7ee44705e"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cad9c5b362e5bbe04167bb926480419e"
server_key_iv: "333616604413f0bda41bae73d692210b"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.384186 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:00.384832 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.384999 7874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:00.385083 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "25fd120315a848a88d8be6c7ee44705e"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cad9c5b362e5bbe04167bb926480419e"
server_key_iv: "333616604413f0bda41bae73d692210b"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.385135 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:00.428630 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:00.431174 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:00.431294 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:00.435755 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43645
I20260925 14:51:00.435868 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43645 every 8 connection(s)
I20260925 14:51:00.436182 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:00.436859 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:00.439124 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Bootstrap starting.
I20260925 14:51:00.439721 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:00.440011 7952 log.cc:824] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:00.440718 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: No bootstrap required, opened a new log
I20260925 14:51:00.442881 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } }
I20260925 14:51:00.443017 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:00.443038 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25fd120315a848a88d8be6c7ee44705e, State: Initialized, Role: FOLLOWER
I20260925 14:51:00.443225 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } }
I20260925 14:51:00.443291 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:00.443312 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:00.443383 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:00.443781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7874
I20260925 14:51:00.443848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:00.443964 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } }
I20260925 14:51:00.444051 392 external_mini_cluster.cc:1428] Setting key e0f3ef9948cf91ca6b4d91b84eaa6bb4
I20260925 14:51:00.444075 7952 leader_election.cc:302] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25fd120315a848a88d8be6c7ee44705e; no voters:
I20260925 14:51:00.444263 7952 leader_election.cc:288] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:00.444294 7955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:00.444454 7955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [term 1 LEADER]: Becoming Leader. State: Replica: 25fd120315a848a88d8be6c7ee44705e, State: Running, Role: LEADER
I20260925 14:51:00.444540 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:00.444589 7955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } }
I20260925 14:51:00.445020 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25fd120315a848a88d8be6c7ee44705e. Latest consensus state: current_term: 1 leader_uuid: "25fd120315a848a88d8be6c7ee44705e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } } }
I20260925 14:51:00.445037 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25fd120315a848a88d8be6c7ee44705e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25fd120315a848a88d8be6c7ee44705e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43645 } } }
I20260925 14:51:00.445207 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:00.445250 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:00.445593 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:00.446089 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:00.447685 7962 catalog_manager.cc:1347] Generated new cluster ID: 9871dc2dbaef4e639cf7ab43e5334a72
I20260925 14:51:00.447749 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:00.454849 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.444755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.0.98.62:43645)
0925 14:51:00.444900 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.444910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.444943 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:51:00.445608 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.445616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.445678 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.445827 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.446709 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.447990 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.448739 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.449174 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.449297 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.451650 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.451654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.451654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.451664 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.453299 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.453663 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.453666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.453667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.453759 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.453965 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.453965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.453966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.454191 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.454345 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.454477 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.454672 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
I20260925 14:51:00.473630 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:00.474432 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:00.483947 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Generated new TSK 0
I20260925 14:51:00.484157 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 14:51:00.515254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43645
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:00.604979 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:00.605221 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:00.605261 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:00.606456 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:00.606523 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:00.606554 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:51:00.606601 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:00.608083 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:00.608348 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:00.608579 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:00.613619 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.613638 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.613611 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.613651 7978 server_base.cc:1034] running on GCE node
I20260925 14:51:00.614077 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:00.614305 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:00.615486 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790347860615475 us; error 28 us; skew 500 ppm
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:00.617892 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:51:00.618319 7978 webserver.cc:460] Webserver started at http://127.0.98.1:39973/ using document root <none> and password file <none>
I20260925 14:51:00.618610 7978 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:00.618657 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:00.618806 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:00.619930 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "34c04634f6849a115cb0f05e439e4893"
server_key_iv: "7f26aa0c66563c726b7aa3c1bd818a0c"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.620293 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "34c04634f6849a115cb0f05e439e4893"
server_key_iv: "7f26aa0c66563c726b7aa3c1bd818a0c"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.621598 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:00.622400 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.622599 7978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:00.622691 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "34c04634f6849a115cb0f05e439e4893"
server_key_iv: "7f26aa0c66563c726b7aa3c1bd818a0c"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.622781 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:00.644801 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:00.647027 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:00.647172 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:00.647709 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:00.647779 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.647822 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:00.647871 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:00.654868 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33125
I20260925 14:51:00.654975 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33125 every 8 connection(s)
I20260925 14:51:00.655330 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:00.657367 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.648199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41091 (local address 127.0.98.62:43645)
0925 14:51:00.648272 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.648274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.648521 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:51:00.649019 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.649021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.649034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.649117 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.650000 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.650897 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.651921 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.652157 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.652249 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.654496 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.654498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.654498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.654507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.656181 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.656533 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.656534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.656534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.656586 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.656830 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.656831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.656831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.656945 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.657011 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.657232 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.657299 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:51:00.657487 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.648185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.1:41091)
0925 14:51:00.648386 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.648426 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.648925 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.649223 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.649226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.649580 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.649878 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.649884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.650996 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.650998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.651814 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.651816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.652272 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.652274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.652399 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.652899 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.652908 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.654407 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:00.656259 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.656263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.656265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.656466 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.656646 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.656647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.656647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.656747 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.657014 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:00.657018 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:00.657123 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 14:51:00.657201 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:51:00.657315 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260925 14:51:00.657972 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43645
I20260925 14:51:00.658056 8109 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:00.658243 8109 heartbeater.cc:511] Master 127.0.98.62:43645 requested a full tablet report, sending...
I20260925 14:51:00.658838 7905 ts_manager.cc:194] Registered new tserver with Master: f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125)
I20260925 14:51:00.659651 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41091
I20260925 14:51:00.660392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 7978
I20260925 14:51:00.660501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:00.660720 392 external_mini_cluster.cc:1428] Setting key 1eea6c1edcaeb03b769ada7469b462b9
I20260925 14:51:00.665409 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.659008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.1:42869)
0925 14:51:00.659122 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.659132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.659189 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.659300 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.659300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.659439 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.659493 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.659494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.660311 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.660312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.661130 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.661132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.661635 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.661636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.661718 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.662122 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.662128 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.662598 (+ 470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:00.664403 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.664404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.664405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.664564 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.664772 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.664773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.664773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.664825 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.665091 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:00.665092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:00.665125 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:51:00.665177 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:51:00.665256 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":69,"threads_started":1}
I20260925 14:51:00.665434 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.659054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42869 (local address 127.0.98.62:43645)
0925 14:51:00.659094 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.659096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.659200 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:51:00.659222 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.659223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.659231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.659297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.659553 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.660212 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.661226 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.661541 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.661609 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.662696 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.662697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.662698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.662703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.664322 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.664638 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.664639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.664640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.664702 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.664888 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.664889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.664889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.665026 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.665097 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.665223 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.665309 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 14:51:00.685379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43645
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:00.765538 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:00.765753 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:00.765776 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:00.766961 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:00.767004 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:00.767024 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:51:00.767041 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:00.768450 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:00.768641 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:00.768821 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:51:00.773962 8118 server_base.cc:1034] running on GCE node
W20260925 14:51:00.773947 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.773976 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.773933 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.774425 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:00.774685 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:00.775856 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790347860775855 us; error 40 us; skew 500 ppm
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:00.778271 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:51:00.778712 8118 webserver.cc:460] Webserver started at http://127.0.98.2:33945/ using document root <none> and password file <none>
I20260925 14:51:00.778949 8118 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:00.779017 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:00.779168 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:00.780197 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d89a18fd88f49a592b99c31f96e3791"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cfe3c830c077fa83675235ec79f0f2e5"
server_key_iv: "20dbd24bc21682eae5ed5c8319fb0746"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.780570 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d89a18fd88f49a592b99c31f96e3791"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cfe3c830c077fa83675235ec79f0f2e5"
server_key_iv: "20dbd24bc21682eae5ed5c8319fb0746"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.781874 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:00.782680 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.782866 8118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:00.782966 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "4d89a18fd88f49a592b99c31f96e3791"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "cfe3c830c077fa83675235ec79f0f2e5"
server_key_iv: "20dbd24bc21682eae5ed5c8319fb0746"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.783044 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:00.815925 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:00.817860 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:00.818025 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:00.818656 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:00.818717 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.818760 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:00.818787 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:00.825863 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34355
I20260925 14:51:00.826128 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34355 every 8 connection(s)
I20260925 14:51:00.826292 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:00.828244 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.819009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45509 (local address 127.0.98.62:43645)
0925 14:51:00.819075 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.819078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.819326 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:51:00.819897 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.819899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.819912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.819961 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.820819 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.821689 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.822850 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.823196 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.823285 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.825577 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.825578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.825579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.825587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.827158 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.827523 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.827524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.827525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.827576 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.827786 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.827786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.827786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.827874 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.827920 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.828124 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.828179 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:51:00.828462 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.819009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.2:45509)
0925 14:51:00.819155 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.819222 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.819745 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.820048 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.820052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.820411 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.820708 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.820714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.821835 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.821837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.822779 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.822781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.823329 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.823331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.823456 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.823965 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.823982 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.825487 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:00.827255 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.827260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.827261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.827468 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.827632 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.827633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.827633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.827746 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.827924 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:00.827928 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:00.828032 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 14:51:00.828111 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:51:00.828221 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 14:51:00.828977 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43645
I20260925 14:51:00.829063 8249 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:00.829270 8249 heartbeater.cc:511] Master 127.0.98.62:43645 requested a full tablet report, sending...
I20260925 14:51:00.829677 7906 ts_manager.cc:194] Registered new tserver with Master: 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355)
I20260925 14:51:00.830120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8118
I20260925 14:51:00.830176 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45509
I20260925 14:51:00.830221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:00.830457 392 external_mini_cluster.cc:1428] Setting key e5c9e21aea5dd0a94d781fc653dad8cf
I20260925 14:51:00.836365 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.829994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57595 (local address 127.0.98.62:43645)
0925 14:51:00.830032 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.830034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.830501 (+ 467us) server_negotiation.cc:407] Connection header received
0925 14:51:00.830569 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.830570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.830580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.830627 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.830989 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.831702 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.832493 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.832672 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.832730 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.833741 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.833742 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.833742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.833748 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.835211 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.835562 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.835563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.835564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.835617 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.835821 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.835822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.835822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.835966 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.836039 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.836194 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.836271 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:00.836381 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.829911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.2:57595)
0925 14:51:00.830430 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.830440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.830512 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.830679 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.830680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.830854 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.830902 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.830903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.831806 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.831807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.832394 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.832397 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.832752 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.832753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.832826 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.833250 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.833256 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.833638 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:00.835303 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.835304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.835305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.835474 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.835690 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.835690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.835691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.835749 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.836041 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:00.836042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:00.836084 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:51:00.836170 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:51:00.836270 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:51:00.857203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43645
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:00.945465 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:00.945715 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:00.945739 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:00.947027 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:00.947069 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:00.947089 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:51:00.947108 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:00.948491 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:00.948669 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:00.948835 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:00.953895 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.953960 8258 server_base.cc:1034] running on GCE node
W20260925 14:51:00.954043 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:00.954176 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:00.954455 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:00.954656 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:00.955793 8258 hybrid_clock.cc:648] HybridClock initialized: now 1790347860955795 us; error 38 us; skew 500 ppm
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:00.958307 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:51:00.958729 8258 webserver.cc:460] Webserver started at http://127.0.98.3:45219/ using document root <none> and password file <none>
I20260925 14:51:00.958901 8258 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:00.958940 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:00.959026 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:00.960075 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "06f4c3165b5740388f2b11d4ddded953"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "d35c93eede8b85ac0ac4410439bd19bf"
server_key_iv: "f73afc1a5ca9fd8620a88f14d225ad71"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.960369 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "06f4c3165b5740388f2b11d4ddded953"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "d35c93eede8b85ac0ac4410439bd19bf"
server_key_iv: "f73afc1a5ca9fd8620a88f14d225ad71"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.961585 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:00.962265 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.962427 8258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:00.962512 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "06f4c3165b5740388f2b11d4ddded953"
format_stamp: "Formatted at 2026-09-25 14:51:00 on dist-test-slave-gpc1"
server_key: "d35c93eede8b85ac0ac4410439bd19bf"
server_key_iv: "f73afc1a5ca9fd8620a88f14d225ad71"
server_key_version: "encryptionkey@0"
I20260925 14:51:00.962564 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:00.980360 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:00.982208 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:00.982319 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:00.982844 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:00.982885 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.982909 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:00.982921 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:00.990054 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34287
I20260925 14:51:00.990114 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34287 every 8 connection(s)
Sep 25 14:51:00 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:00.990490 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:00.992419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8258
I20260925 14:51:00.992523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:00.992767 392 external_mini_cluster.cc:1428] Setting key f976b9c4f4a1af8620ee6b2e13973395
I20260925 14:51:00.994163 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.983403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39953 (local address 127.0.98.62:43645)
0925 14:51:00.983475 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.983478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.983639 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:51:00.984467 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.984468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.984481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.984552 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.985531 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.986405 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.987839 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.988179 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.988274 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.991038 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.991040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.991040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.991045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:00.992528 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.993069 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.993070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.993070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.993156 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:00.993490 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:00.993491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:00.993491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:00.993635 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:00.993712 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:00.994031 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 14:51:00.994098 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:51:00.994333 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.983363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.3:39953)
0925 14:51:00.983510 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.983549 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.984307 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.984696 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.984703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.985106 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.985417 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.985425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.986497 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.986498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.987755 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.987758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.988282 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.988283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.988417 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.988990 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.989002 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.990947 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:00.992642 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.992650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.992652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.992974 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.993265 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:00.993266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:00.993267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:00.993416 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:00.993712 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:00.993717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:00.993890 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:51:00.993983 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:51:00.994124 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
I20260925 14:51:00.994797 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43645
I20260925 14:51:00.994886 8389 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:00.995095 8389 heartbeater.cc:511] Master 127.0.98.62:43645 requested a full tablet report, sending...
I20260925 14:51:00.995499 7906 ts_manager.cc:194] Registered new tserver with Master: 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:00.995867 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39953
I20260925 14:51:01.001528 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.996238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58615 (local address 127.0.98.62:43645)
0925 14:51:00.996287 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:00.996289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:00.996401 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:51:00.996486 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:00.996487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:00.996497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:00.996552 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:00.996875 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.997348 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.998044 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.998252 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.998324 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.999199 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:00.999201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:00.999201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:00.999208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:01.000441 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.000796 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.000798 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.000798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.000854 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.001049 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.001050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.001050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.001156 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:01.001243 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.001368 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.001437 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:01.001549 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:00.996154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.98.3:58615)
0925 14:51:00.996325 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:00.996335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:00.996429 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:00.996556 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:00.996557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:00.996746 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:00.996799 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.996800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.997441 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.997442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.997959 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:00.997960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:00.998334 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:00.998335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:00.998440 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:00.998846 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:00.998850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:00.999110 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:01.000526 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:01.000527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:01.000528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:01.000698 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:01.000919 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:01.000920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:01.000921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:01.000977 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:01.001227 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:01.001228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:01.001270 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.001338 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.001439 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260925 14:51:01.004528 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:51:01.025128 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:01.019808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.0.98.62:43645)
0925 14:51:01.019876 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.019878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.019888 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:01.019927 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.019928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.019936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.019986 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:01.020530 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.021023 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.021747 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.021930 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.021987 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.025040 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33564: BlockingRecv error: recv got EOF from 127.0.0.1:33564 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:01.032650 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.025993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.98.62:43645)
0925 14:51:01.026061 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.026063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.026150 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:51:01.026248 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.026249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.026258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.026307 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:01.026821 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.027294 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.027967 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.028165 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.028258 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.030483 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:01.030484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:01.030484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:01.030490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:01.031660 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.031963 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.031964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.031964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.032014 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.032267 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.032268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.032268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.032361 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:01.032433 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.032530 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.032586 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:51:01.034204 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:01.035837 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:01.042502 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33125 (local address 127.0.0.1:57994)
0925 14:51:01.038206 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.038263 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.038317 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.038678 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.038681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.038689 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.038766 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.038769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.040211 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.040212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.041266 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.041268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042177 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.042259 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.042264 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.042296 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.042389 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 14:51:01.042500 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.98.3:34287)
0925 14:51:01.038569 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.038575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.038587 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:01.038675 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.038678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.038698 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.038830 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.039092 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.039812 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.041292 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042069 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042145 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.042231 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.042318 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.042389 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":92,"threads_started":1}
I20260925 14:51:01.042753 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.0.98.1:33125)
0925 14:51:01.038523 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.038529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.038540 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:01.038580 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.038583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.038601 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.038731 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.038960 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.040136 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.041354 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042069 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042486 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.042570 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.042597 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.042642 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260925 14:51:01.043219 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.0.1:60420)
0925 14:51:01.038485 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.038495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.038540 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.038877 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.038878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.038880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.038927 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.038928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.041354 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.041355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.042009 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.042010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042985 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042985 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.043030 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.043034 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.043114 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.043157 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":69,"threads_started":1}
I20260925 14:51:01.043300 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.0.98.2:34355)
0925 14:51:01.038675 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.038679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.038688 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:01.038724 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.038726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.038741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.038813 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.038994 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.040120 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042106 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042900 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042979 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.043064 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.043148 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.043188 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260925 14:51:01.043512 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.038046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34287 (local address 127.0.0.1:44092)
0925 14:51:01.038204 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.038248 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.038312 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.038828 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.038829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.038841 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.038878 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.038880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.039901 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.039902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.041195 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.041197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.042145 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.042146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.042223 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.042238 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.042306 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.042435 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":114,"thread_start_us":60,"threads_started":1}
I20260925 14:51:01.043555 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 865115c76e834cc6bf5dcb39868c30ba (DEFAULT_TABLE table=test-table [id=0968e6c6a40544dba52a62766542db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.043939 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865115c76e834cc6bf5dcb39868c30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.043926 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 865115c76e834cc6bf5dcb39868c30ba (DEFAULT_TABLE table=test-table [id=0968e6c6a40544dba52a62766542db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.044245 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865115c76e834cc6bf5dcb39868c30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.046785 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 865115c76e834cc6bf5dcb39868c30ba (DEFAULT_TABLE table=test-table [id=0968e6c6a40544dba52a62766542db88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.047120 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 865115c76e834cc6bf5dcb39868c30ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.047449 8417 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Bootstrap starting.
I20260925 14:51:01.047600 8418 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Bootstrap starting.
I20260925 14:51:01.048158 8418 tablet_bootstrap.cc:654] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.048158 8417 tablet_bootstrap.cc:654] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.048501 8417 log.cc:824] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:01.048501 8418 log.cc:824] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:01.049252 8418 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: No bootstrap required, opened a new log
I20260925 14:51:01.049322 8418 ts_tablet_manager.cc:1397] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:01.049445 8417 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: No bootstrap required, opened a new log
I20260925 14:51:01.049504 8417 ts_tablet_manager.cc:1397] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:01.050388 8421 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Bootstrap starting.
I20260925 14:51:01.051015 8421 tablet_bootstrap.cc:654] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.051318 8421 log.cc:824] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:01.051365 8418 raft_consensus.cc:348] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.051548 8418 raft_consensus.cc:374] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.051594 8418 raft_consensus.cc:729] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.051729 8418 consensus_queue.cc:260] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.051705 8417 raft_consensus.cc:348] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.051891 8417 raft_consensus.cc:374] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.051944 8417 raft_consensus.cc:729] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.051996 8418 ts_tablet_manager.cc:1428] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:01.052055 8417 consensus_queue.cc:260] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.052127 8249 heartbeater.cc:503] Master 127.0.98.62:43645 was elected leader, sending a full tablet report...
I20260925 14:51:01.052176 8421 tablet_bootstrap.cc:492] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: No bootstrap required, opened a new log
I20260925 14:51:01.052234 8421 ts_tablet_manager.cc:1397] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:01.052362 8417 ts_tablet_manager.cc:1428] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:01.052493 8109 heartbeater.cc:503] Master 127.0.98.62:43645 was elected leader, sending a full tablet report...
I20260925 14:51:01.054060 8421 raft_consensus.cc:348] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.054205 8421 raft_consensus.cc:374] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.054250 8421 raft_consensus.cc:729] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.054394 8421 consensus_queue.cc:260] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.054672 8421 ts_tablet_manager.cc:1428] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:51:01.054708 8389 heartbeater.cc:503] Master 127.0.98.62:43645 was elected leader, sending a full tablet report...
I20260925 14:51:01.056558 8423 raft_consensus.cc:482] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:01.056622 8423 raft_consensus.cc:504] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.056962 8423 leader_election.cc:288] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287), f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125)
I20260925 14:51:01.059733 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.057163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49725 (local address 127.0.98.3:34287)
0925 14:51:01.057222 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.057239 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.057249 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:01.057334 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.057335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.057346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.057395 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.057520 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.057977 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.058978 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.059391 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.059469 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.059479 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.059644 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.059677 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:51:01.059753 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.057106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34287 (local address 127.0.98.2:49725)
0925 14:51:01.057163 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.057176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.057218 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.057398 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.057399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.057400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.057455 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.057457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.058056 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.058057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.058883 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.058885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.059467 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.059468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.059551 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.059554 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.059629 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.059670 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:51:01.059891 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.057289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35315 (local address 127.0.98.1:33125)
0925 14:51:01.057325 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.057327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.057427 (+ 100us) server_negotiation.cc:407] Connection header received
0925 14:51:01.057488 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.057489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.057500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.057554 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.057696 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.058232 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.059186 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.059537 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.059627 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.059636 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.059799 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.059833 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:51:01.059909 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.057223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33125 (local address 127.0.98.2:35315)
0925 14:51:01.057344 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.057353 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.057439 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.057557 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.057558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.057559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.057624 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.057626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.058320 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.058321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.059100 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.059102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.059631 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.059632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.059701 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.059705 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.059791 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.059828 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260925 14:51:01.060204 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865115c76e834cc6bf5dcb39868c30ba" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953" is_pre_election: true
I20260925 14:51:01.060271 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865115c76e834cc6bf5dcb39868c30ba" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" is_pre_election: true
I20260925 14:51:01.060321 8343 raft_consensus.cc:2454] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.060366 8063 raft_consensus.cc:2454] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.060555 8136 leader_election.cc:302] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4c3165b5740388f2b11d4ddded953, 4d89a18fd88f49a592b99c31f96e3791; no voters:
I20260925 14:51:01.060673 8423 raft_consensus.cc:2781] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:01.060731 8423 raft_consensus.cc:482] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:01.060765 8423 raft_consensus.cc:3037] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.061599 8423 raft_consensus.cc:504] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.061743 8423 leader_election.cc:288] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Requested vote from peers 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287), f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125)
I20260925 14:51:01.061908 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865115c76e834cc6bf5dcb39868c30ba" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953"
I20260925 14:51:01.061936 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "865115c76e834cc6bf5dcb39868c30ba" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
I20260925 14:51:01.061985 8343 raft_consensus.cc:3037] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.061995 8063 raft_consensus.cc:3037] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.062831 8063 raft_consensus.cc:2454] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.062834 8343 raft_consensus.cc:2454] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.063019 8137 leader_election.cc:302] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.063129 8423 raft_consensus.cc:2781] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:01.063271 8423 raft_consensus.cc:686] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 1 LEADER]: Becoming Leader. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Running, Role: LEADER
I20260925 14:51:01.063372 8423 consensus_queue.cc:237] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.064105 7906 catalog_manager.cc:5495] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } health_report { overall_health: HEALTHY } } }
I20260925 14:51:01.070267 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.068051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.0.98.2:34355)
0925 14:51:01.068103 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.068104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.068113 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:01.068166 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.068167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.068183 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.068243 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.068343 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.068825 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.069534 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.069738 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.069804 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.069932 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.070022 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.070083 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.070124 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.070141 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:51:01.072021 8063 raft_consensus.cc:1261] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.072078 8343 raft_consensus.cc:1261] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.072324 8423 consensus_queue.cc:1035] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.072482 8427 consensus_queue.cc:1035] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.074954 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333264838948749312 to 7333264838916173824. Current Snapshot: MvccSnapshot[applied={T|T < 7333264838948749312}]
I20260925 14:51:01.077087 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.077183 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.077858 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Sending DeleteTablet for 3 replicas of tablet 865115c76e834cc6bf5dcb39868c30ba
I20260925 14:51:01.078194 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 865115c76e834cc6bf5dcb39868c30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60420
I20260925 14:51:01.078195 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 865115c76e834cc6bf5dcb39868c30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260925 14:51:01.078475 8440 tablet_replica.cc:330] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: stopping tablet replica
I20260925 14:51:01.078464 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 865115c76e834cc6bf5dcb39868c30ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57994
I20260925 14:51:01.078580 8440 raft_consensus.cc:2229] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.078702 8440 raft_consensus.cc:2258] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.078648 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:01.078779 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:01.078698 8441 tablet_replica.cc:330] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: stopping tablet replica
I20260925 14:51:01.079010 8441 raft_consensus.cc:2229] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.079134 8440 ts_tablet_manager.cc:1905] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.079156 8441 raft_consensus.cc:2258] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.080068 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (DEFAULT_TABLE table=test-table [id=46588975df4e4b9ab837e032e87e7214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.080107 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (DEFAULT_TABLE table=test-table [id=46588975df4e4b9ab837e032e87e7214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.080183 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b32a13639b40eb84ca02ff5dd9a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.080199 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b32a13639b40eb84ca02ff5dd9a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.081504 8421 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Bootstrap starting.
I20260925 14:51:01.081815 8417 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Bootstrap starting.
I20260925 14:51:01.081952 8421 tablet_bootstrap.cc:654] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.082233 8417 tablet_bootstrap.cc:654] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.082693 8421 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: No bootstrap required, opened a new log
I20260925 14:51:01.082749 8421 ts_tablet_manager.cc:1397] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.082741 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (DEFAULT_TABLE table=test-table [id=46588975df4e4b9ab837e032e87e7214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.082844 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b32a13639b40eb84ca02ff5dd9a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.082885 8421 raft_consensus.cc:348] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.082944 8421 raft_consensus.cc:374] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.082964 8421 raft_consensus.cc:729] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.083032 8421 consensus_queue.cc:260] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.083112 8417 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: No bootstrap required, opened a new log
I20260925 14:51:01.083114 8421 ts_tablet_manager.cc:1428] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.083148 8417 ts_tablet_manager.cc:1397] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.083313 8417 raft_consensus.cc:348] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.083377 8417 raft_consensus.cc:374] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.083395 8417 raft_consensus.cc:729] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.083432 8417 consensus_queue.cc:260] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.083510 8417 ts_tablet_manager.cc:1428] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.082970 8440 ts_tablet_manager.cc:1918] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.084054 8440 log.cc:1196] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/865115c76e834cc6bf5dcb39868c30ba
I20260925 14:51:01.084306 8440 ts_tablet_manager.cc:1939] T 865115c76e834cc6bf5dcb39868c30ba P 06f4c3165b5740388f2b11d4ddded953: Deleting consensus metadata
I20260925 14:51:01.084323 8418 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Bootstrap starting.
I20260925 14:51:01.084704 8418 tablet_bootstrap.cc:654] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.084733 7892 catalog_manager.cc:4841] TS 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287): tablet 865115c76e834cc6bf5dcb39868c30ba (table test-table [id=0968e6c6a40544dba52a62766542db88]) successfully deleted
I20260925 14:51:01.085323 8418 tablet_bootstrap.cc:492] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: No bootstrap required, opened a new log
I20260925 14:51:01.085377 8418 ts_tablet_manager.cc:1397] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.085430 8441 ts_tablet_manager.cc:1905] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.085505 8418 raft_consensus.cc:348] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.085562 8418 raft_consensus.cc:374] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.085583 8418 raft_consensus.cc:729] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.085628 8418 consensus_queue.cc:260] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.085721 8418 ts_tablet_manager.cc:1428] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.086473 8439 tablet_replica.cc:330] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: stopping tablet replica
I20260925 14:51:01.086566 8439 raft_consensus.cc:2229] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:01.086666 8439 raft_consensus.cc:2258] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.087023 8441 ts_tablet_manager.cc:1918] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.087056 8439 ts_tablet_manager.cc:1905] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.087088 8441 log.cc:1196] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/865115c76e834cc6bf5dcb39868c30ba
I20260925 14:51:01.087342 8441 ts_tablet_manager.cc:1939] T 865115c76e834cc6bf5dcb39868c30ba P f7d12dcbacbb4c418f740f17b7fc6671: Deleting consensus metadata
I20260925 14:51:01.087754 7893 catalog_manager.cc:4841] TS f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125): tablet 865115c76e834cc6bf5dcb39868c30ba (table test-table [id=0968e6c6a40544dba52a62766542db88]) successfully deleted
I20260925 14:51:01.088575 8439 ts_tablet_manager.cc:1918] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.088666 8439 log.cc:1196] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/865115c76e834cc6bf5dcb39868c30ba
I20260925 14:51:01.088939 8439 ts_tablet_manager.cc:1939] T 865115c76e834cc6bf5dcb39868c30ba P 4d89a18fd88f49a592b99c31f96e3791: Deleting consensus metadata
I20260925 14:51:01.089306 7893 catalog_manager.cc:4841] TS 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355): tablet 865115c76e834cc6bf5dcb39868c30ba (table test-table [id=0968e6c6a40544dba52a62766542db88]) successfully deleted
W20260925 14:51:01.156144 8110 tablet.cc:2369] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:01.163056 8425 raft_consensus.cc:482] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:01.163158 8425 raft_consensus.cc:504] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.163475 8425 leader_election.cc:288] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355)
I20260925 14:51:01.167482 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.163676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33125 (local address 127.0.98.3:44823)
0925 14:51:01.163769 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.163784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.163852 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.164082 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.164083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.164085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.164183 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.164185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.165168 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.165170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.166455 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.166458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.167150 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.167151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.167253 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.167260 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.167353 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.167402 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":7}
I20260925 14:51:01.167480 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.163810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44823 (local address 127.0.98.1:33125)
0925 14:51:01.163858 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.163860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.163871 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:01.163961 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.163962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.163975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.164030 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.164281 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.165059 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.166580 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.166980 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.167110 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.167122 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.167363 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.167405 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:51:01.167518 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.163945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43037 (local address 127.0.98.2:34355)
0925 14:51:01.163989 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.163991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.164003 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:01.164072 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.164073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.164081 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.164140 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.164328 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.164845 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.166540 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.167109 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.167182 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.167191 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.167412 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.167450 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:01.167804 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.163783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.98.3:43037)
0925 14:51:01.163949 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.163960 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.164011 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.164142 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.164143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.164145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.164219 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.164220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.164942 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.164943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.166455 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.166458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.167248 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.167250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.167331 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.167335 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.167630 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.167684 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260925 14:51:01.167928 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b32a13639b40eb84ca02ff5dd9a2c6" candidate_uuid: "06f4c3165b5740388f2b11d4ddded953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" is_pre_election: true
I20260925 14:51:01.168025 8063 raft_consensus.cc:2454] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f4c3165b5740388f2b11d4ddded953 in term 0.
I20260925 14:51:01.168177 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b32a13639b40eb84ca02ff5dd9a2c6" candidate_uuid: "06f4c3165b5740388f2b11d4ddded953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89a18fd88f49a592b99c31f96e3791" is_pre_election: true
I20260925 14:51:01.168289 8203 raft_consensus.cc:2454] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f4c3165b5740388f2b11d4ddded953 in term 0.
I20260925 14:51:01.168232 8277 leader_election.cc:302] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4c3165b5740388f2b11d4ddded953, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.168391 8425 raft_consensus.cc:2781] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:01.168426 8425 raft_consensus.cc:482] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:01.168447 8425 raft_consensus.cc:3037] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.169381 8425 raft_consensus.cc:504] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.169519 8425 leader_election.cc:288] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [CANDIDATE]: Term 1 election: Requested vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355)
I20260925 14:51:01.169694 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b32a13639b40eb84ca02ff5dd9a2c6" candidate_uuid: "06f4c3165b5740388f2b11d4ddded953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
I20260925 14:51:01.169752 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b32a13639b40eb84ca02ff5dd9a2c6" candidate_uuid: "06f4c3165b5740388f2b11d4ddded953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89a18fd88f49a592b99c31f96e3791"
I20260925 14:51:01.169804 8063 raft_consensus.cc:3037] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.169844 8203 raft_consensus.cc:3037] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.170553 8063 raft_consensus.cc:2454] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f4c3165b5740388f2b11d4ddded953 in term 1.
I20260925 14:51:01.170701 8203 raft_consensus.cc:2454] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f4c3165b5740388f2b11d4ddded953 in term 1.
I20260925 14:51:01.170734 8277 leader_election.cc:302] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4c3165b5740388f2b11d4ddded953, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.170885 8425 raft_consensus.cc:2781] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:01.171032 8425 raft_consensus.cc:686] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 LEADER]: Becoming Leader. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Running, Role: LEADER
I20260925 14:51:01.171106 8425 consensus_queue.cc:237] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.171833 7906 catalog_manager.cc:5495] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "06f4c3165b5740388f2b11d4ddded953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } health_report { overall_health: HEALTHY } } }
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347860, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:01.217509 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.211026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43645 (local address 127.0.0.1:33594)
0925 14:51:01.211212 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.211225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.211318 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.211462 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.211463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.211793 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:01.211836 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.211839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.212507 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.212508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.213030 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.213031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.213396 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.213397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.213458 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.213930 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:01.213935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:01.215252 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:01.216595 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:01.216600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:01.216602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:01.216788 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:01.216952 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:01.216952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:01.216953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:01.217005 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:01.217215 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:01.217220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:01.217264 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.217313 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.217397 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260925 14:51:01.217521 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.211111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.0.98.62:43645)
0925 14:51:01.211184 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.211186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.211305 (+ 119us) server_negotiation.cc:407] Connection header received
0925 14:51:01.211373 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.211374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.211384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.211458 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:01.211927 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.212423 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.213117 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.213301 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.213394 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.215335 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:01.215336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:01.215337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:01.215343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:01.216526 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.216850 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.216851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.216851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.216897 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.217055 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.217055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.217055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.217146 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:01.217217 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.217331 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.217440 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":45}
I20260925 14:51:01.219432 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:51:01.219632 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:01.221007 8323 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb97ed5d324852b1895c87028300f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726d6f82c3a243a7a9ebc98a8c63410c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:51:01.221035 8181 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb97ed5d324852b1895c87028300f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726d6f82c3a243a7a9ebc98a8c63410c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:51:01.221167 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb97ed5d324852b1895c87028300f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.221167 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb97ed5d324852b1895c87028300f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.222683 8421 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: Bootstrap starting.
I20260925 14:51:01.222766 8043 tablet_service.cc:1467] Processing CreateTablet for tablet ccdb97ed5d324852b1895c87028300f6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=726d6f82c3a243a7a9ebc98a8c63410c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:51:01.222891 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdb97ed5d324852b1895c87028300f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.223093 8421 tablet_bootstrap.cc:654] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.223383 8418 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: Bootstrap starting.
I20260925 14:51:01.223670 8418 tablet_bootstrap.cc:654] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.223749 8421 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: No bootstrap required, opened a new log
I20260925 14:51:01.223793 8421 ts_tablet_manager.cc:1397] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.223960 8421 raft_consensus.cc:348] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.224035 8421 raft_consensus.cc:374] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.224066 8421 raft_consensus.cc:729] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.224113 8421 consensus_queue.cc:260] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.224133 8417 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: Bootstrap starting.
I20260925 14:51:01.224221 8421 ts_tablet_manager.cc:1428] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:01.224244 8418 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: No bootstrap required, opened a new log
I20260925 14:51:01.224282 8418 ts_tablet_manager.cc:1397] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.224293 8425 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.224386 8425 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.224380 8418 raft_consensus.cc:348] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.224436 8418 raft_consensus.cc:374] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.224463 8418 raft_consensus.cc:729] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.224502 8417 tablet_bootstrap.cc:654] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.224520 8418 consensus_queue.cc:260] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.224620 8418 ts_tablet_manager.cc:1428] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.224637 8423 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.224717 8423 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.225333 8417 tablet_bootstrap.cc:492] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: No bootstrap required, opened a new log
I20260925 14:51:01.225389 8417 ts_tablet_manager.cc:1397] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.225538 8417 raft_consensus.cc:348] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.225617 8417 raft_consensus.cc:374] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.225643 8417 raft_consensus.cc:729] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.225693 8417 consensus_queue.cc:260] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.225790 8417 ts_tablet_manager.cc:1428] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.225842 8424 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.225915 8424 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:51:01.241117 8390 tablet.cc:2369] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:51:01.327702 8250 tablet.cc:2369] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:01.360369 8423 raft_consensus.cc:482] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:01.360484 8423 raft_consensus.cc:504] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.360813 8423 leader_election.cc:288] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.361049 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb97ed5d324852b1895c87028300f6" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" is_pre_election: true
I20260925 14:51:01.361088 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb97ed5d324852b1895c87028300f6" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953" is_pre_election: true
I20260925 14:51:01.361187 8063 raft_consensus.cc:2454] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.361228 8343 raft_consensus.cc:2454] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.361377 8137 leader_election.cc:302] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.361532 8423 raft_consensus.cc:2781] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:01.361586 8423 raft_consensus.cc:482] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:01.361619 8423 raft_consensus.cc:3037] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.362289 8423 raft_consensus.cc:504] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.362485 8423 leader_election.cc:288] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Requested vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.362654 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb97ed5d324852b1895c87028300f6" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953"
I20260925 14:51:01.362720 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdb97ed5d324852b1895c87028300f6" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
I20260925 14:51:01.362779 8343 raft_consensus.cc:3037] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.362776 8063 raft_consensus.cc:3037] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.363476 8063 raft_consensus.cc:2454] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.363648 8137 leader_election.cc:302] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.363755 8423 raft_consensus.cc:2781] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:01.363806 8423 raft_consensus.cc:686] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 LEADER]: Becoming Leader. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Running, Role: LEADER
I20260925 14:51:01.363859 8423 consensus_queue.cc:237] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } }
I20260925 14:51:01.363981 8343 raft_consensus.cc:2454] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.364174 8427 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: TxnStatusTablet state changed. Reason: New leader 4d89a18fd88f49a592b99c31f96e3791. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.364296 8427 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:51:01.364470 8458 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:51:01.364610 8458 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:51:01.364842 7905 catalog_manager.cc:5495] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } health_report { overall_health: HEALTHY } } }
I20260925 14:51:01.370321 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.367822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.98.2:34355)
0925 14:51:01.367874 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.367876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.367886 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:01.367938 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.367939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.367951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.368020 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.368119 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.368834 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.369581 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.369876 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.369948 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.370089 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.370122 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.370176 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.370233 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.370253 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:51:01.370322 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.367738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.0.1:60678)
0925 14:51:01.367810 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.367822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.367868 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.368023 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.368023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.368024 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:51:01.368080 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.368082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.368936 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.368937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.369500 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.369501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.369951 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.369952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.369999 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.370012 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.370187 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.370193 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.370229 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.370268 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:51:01.371268 8343 raft_consensus.cc:1261] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.371507 8423 consensus_queue.cc:1035] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.371546 8063 raft_consensus.cc:1261] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.371745 8423 consensus_queue.cc:1035] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.372027 8425 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: TxnStatusTablet state changed. Reason: New leader 4d89a18fd88f49a592b99c31f96e3791. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.372105 8425 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.372690 8424 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: TxnStatusTablet state changed. Reason: New leader 4d89a18fd88f49a592b99c31f96e3791. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.372784 8424 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.372816 8423 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.372891 8423 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:51:01.373489 8425 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.373565 8425 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 06f4c3165b5740388f2b11d4ddded953: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.374100 8427 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.374177 8427 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P 4d89a18fd88f49a592b99c31f96e3791: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:51:01.374782 8424 tablet_replica.cc:439] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } }
I20260925 14:51:01.374866 8424 tablet_replica.cc:442] T ccdb97ed5d324852b1895c87028300f6 P f7d12dcbacbb4c418f740f17b7fc6671: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:51:01.377127 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.374574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.0.98.3:34287)
0925 14:51:01.374611 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.374613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.374621 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:01.374654 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.374655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.374686 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.374751 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.375030 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.375474 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.376367 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.376577 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.376650 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.376784 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.376903 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.376983 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.377036 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.377056 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:51:01.382315 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.379701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42789 (local address 127.0.98.2:34355)
0925 14:51:01.379755 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.379757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.379764 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:51:01.379781 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.379782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.379791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.379838 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.380006 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.380461 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.381606 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.381785 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.381836 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.382073 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.382101 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.382159 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.382249 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.382267 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:51:01.382403 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.379458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.98.3:42789)
0925 14:51:01.379539 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.379551 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.379591 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.379866 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.379867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.379868 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:51:01.379934 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.379936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.380534 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.380535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.381519 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.381521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.381859 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.381861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.381934 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.381950 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.382180 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.382182 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.382249 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.382297 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 14:51:01.384742 8063 raft_consensus.cc:1261] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Refusing update from remote peer 06f4c3165b5740388f2b11d4ddded953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.384835 8203 raft_consensus.cc:1261] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Refusing update from remote peer 06f4c3165b5740388f2b11d4ddded953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.385121 8425 consensus_queue.cc:1035] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.385238 8446 consensus_queue.cc:1035] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.386411 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333264840230121472 to 7333264839357509632. Current Snapshot: MvccSnapshot[applied={T|T < 7333264840230121472}]
I20260925 14:51:01.386831 8431 mvcc.cc:205] Tried to move back new op lower bound from 7333264840230121472 to 7333264839357509632. Current Snapshot: MvccSnapshot[applied={T|T < 7333264840230121472}]
I20260925 14:51:01.393826 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.391015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46061 (local address 127.0.98.3:34287)
0925 14:51:01.391069 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.391071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.391199 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:51:01.391220 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.391222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.391232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.391279 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.391408 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.391866 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.393030 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.393356 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.393422 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.393634 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.393659 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.393695 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.393763 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.393779 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:51:01.393893 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.390887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34287 (local address 127.0.98.2:46061)
0925 14:51:01.391065 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.391079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.391152 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.391295 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.391295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.391297 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:51:01.391348 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.391350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.391945 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.391946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.392931 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.392934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.393444 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.393445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.393510 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.393529 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.393700 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.393701 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.393764 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.393809 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260925 14:51:01.404170 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.404280 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.404929 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Sending DeleteTablet for 3 replicas of tablet 20b32a13639b40eb84ca02ff5dd9a2c6
I20260925 14:51:01.405179 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57994
I20260925 14:51:01.405278 8441 tablet_replica.cc:330] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: stopping tablet replica
I20260925 14:51:01.405287 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260925 14:51:01.405326 8441 raft_consensus.cc:2229] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.405359 8440 tablet_replica.cc:330] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: stopping tablet replica
I20260925 14:51:01.405396 8441 raft_consensus.cc:2258] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.405397 8440 raft_consensus.cc:2229] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:01.405484 8440 raft_consensus.cc:2258] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.405814 8441 ts_tablet_manager.cc:1905] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.405912 8440 ts_tablet_manager.cc:1905] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.407617 8441 ts_tablet_manager.cc:1918] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:51:01.407692 8441 log.cc:1196] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/20b32a13639b40eb84ca02ff5dd9a2c6
I20260925 14:51:01.407907 8441 ts_tablet_manager.cc:1939] T 20b32a13639b40eb84ca02ff5dd9a2c6 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting consensus metadata
I20260925 14:51:01.408584 7893 catalog_manager.cc:4841] TS f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125): tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (table test-table [id=46588975df4e4b9ab837e032e87e7214]) successfully deleted
I20260925 14:51:01.410194 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b32a13639b40eb84ca02ff5dd9a2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60420
I20260925 14:51:01.410218 8440 ts_tablet_manager.cc:1918] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:51:01.410280 8440 log.cc:1196] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/20b32a13639b40eb84ca02ff5dd9a2c6
I20260925 14:51:01.410516 8440 ts_tablet_manager.cc:1939] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 06f4c3165b5740388f2b11d4ddded953: Deleting consensus metadata
I20260925 14:51:01.410763 8439 tablet_replica.cc:330] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: stopping tablet replica
I20260925 14:51:01.410838 8439 raft_consensus.cc:2229] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.410892 8439 raft_consensus.cc:2258] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.411121 7892 catalog_manager.cc:4841] TS 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287): tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (table test-table [id=46588975df4e4b9ab837e032e87e7214]) successfully deleted
I20260925 14:51:01.411329 8439 ts_tablet_manager.cc:1905] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.412853 8439 ts_tablet_manager.cc:1918] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:51:01.412914 8439 log.cc:1196] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/20b32a13639b40eb84ca02ff5dd9a2c6
I20260925 14:51:01.413118 8439 ts_tablet_manager.cc:1939] T 20b32a13639b40eb84ca02ff5dd9a2c6 P 4d89a18fd88f49a592b99c31f96e3791: Deleting consensus metadata
I20260925 14:51:01.413465 7893 catalog_manager.cc:4841] TS 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355): tablet 20b32a13639b40eb84ca02ff5dd9a2c6 (table test-table [id=46588975df4e4b9ab837e032e87e7214]) successfully deleted
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:51:01.440726 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:01.433778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.98.62:43645)
0925 14:51:01.433828 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.433830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.433985 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:51:01.434054 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.434056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.434068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.434129 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:01.434921 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.435702 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.436836 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.437128 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.437179 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.440665 (+ 3486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55858: BlockingRecv error: recv got EOF from 127.0.0.1:55858 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:51:01.449965 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.441491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.98.62:43645)
0925 14:51:01.441526 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.441527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.441753 (+ 226us) server_negotiation.cc:407] Connection header received
0925 14:51:01.441825 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.441826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.441836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.441893 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:01.442473 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.443204 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.444250 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.444537 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.444608 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.447194 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:01.447196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:01.447196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:01.447203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:01.448790 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.449198 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.449199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.449200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.449258 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:01.449463 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:01.449464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:01.449464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:01.449586 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:01.449677 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.449806 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.449880 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:51:01.451912 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:01.452090 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:01.453447 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 (DEFAULT_TABLE table=test-table [id=0e4b39212273487da25d68c3896241ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.453482 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 (DEFAULT_TABLE table=test-table [id=0e4b39212273487da25d68c3896241ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.453573 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f057673a84c4127960531c3d5f1fdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.453606 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f057673a84c4127960531c3d5f1fdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.454454 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 (DEFAULT_TABLE table=test-table [id=0e4b39212273487da25d68c3896241ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.454564 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f057673a84c4127960531c3d5f1fdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.454917 8418 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Bootstrap starting.
I20260925 14:51:01.455219 8418 tablet_bootstrap.cc:654] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.455763 8417 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Bootstrap starting.
I20260925 14:51:01.456151 8417 tablet_bootstrap.cc:654] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.456810 8417 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: No bootstrap required, opened a new log
I20260925 14:51:01.456858 8417 ts_tablet_manager.cc:1397] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.457001 8417 raft_consensus.cc:348] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.457088 8417 raft_consensus.cc:374] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.457113 8417 raft_consensus.cc:729] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.457170 8417 consensus_queue.cc:260] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.457202 8421 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Bootstrap starting.
I20260925 14:51:01.457278 8417 ts_tablet_manager.cc:1428] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.457616 8421 tablet_bootstrap.cc:654] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.458451 8421 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: No bootstrap required, opened a new log
I20260925 14:51:01.458495 8421 ts_tablet_manager.cc:1397] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.458626 8421 raft_consensus.cc:348] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.458694 8421 raft_consensus.cc:374] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.458715 8421 raft_consensus.cc:729] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.458751 8418 tablet_bootstrap.cc:492] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: No bootstrap required, opened a new log
I20260925 14:51:01.458755 8421 consensus_queue.cc:260] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.458796 8418 ts_tablet_manager.cc:1397] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 14:51:01.458828 8421 ts_tablet_manager.cc:1428] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.458885 8418 raft_consensus.cc:348] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.458950 8418 raft_consensus.cc:374] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.458969 8418 raft_consensus.cc:729] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.459005 8418 consensus_queue.cc:260] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.459075 8418 ts_tablet_manager.cc:1428] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.584678 8465 raft_consensus.cc:482] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:01.584791 8465 raft_consensus.cc:504] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.585206 8465 leader_election.cc:288] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.585421 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f057673a84c4127960531c3d5f1fdf2" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" is_pre_election: true
I20260925 14:51:01.585450 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f057673a84c4127960531c3d5f1fdf2" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953" is_pre_election: true
I20260925 14:51:01.585541 8063 raft_consensus.cc:2454] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.585590 8343 raft_consensus.cc:2454] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 0.
I20260925 14:51:01.585742 8137 leader_election.cc:302] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.585839 8465 raft_consensus.cc:2781] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:01.585878 8465 raft_consensus.cc:482] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:01.585907 8465 raft_consensus.cc:3037] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.586712 8465 raft_consensus.cc:504] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.586886 8465 leader_election.cc:288] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Requested vote from peers f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.587054 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f057673a84c4127960531c3d5f1fdf2" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7d12dcbacbb4c418f740f17b7fc6671"
I20260925 14:51:01.587118 8063 raft_consensus.cc:3037] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.587735 8063 raft_consensus.cc:2454] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.587862 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f057673a84c4127960531c3d5f1fdf2" candidate_uuid: "4d89a18fd88f49a592b99c31f96e3791" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953"
I20260925 14:51:01.587903 8137 leader_election.cc:302] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.587946 8343 raft_consensus.cc:3037] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.588013 8465 raft_consensus.cc:2781] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:01.588085 8465 raft_consensus.cc:686] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 LEADER]: Becoming Leader. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Running, Role: LEADER
I20260925 14:51:01.588137 8465 consensus_queue.cc:237] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.588654 8343 raft_consensus.cc:2454] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d89a18fd88f49a592b99c31f96e3791 in term 1.
I20260925 14:51:01.588753 7905 catalog_manager.cc:5495] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4d89a18fd88f49a592b99c31f96e3791" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } health_report { overall_health: UNKNOWN } } }
I20260925 14:51:01.605942 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.603822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.98.2:34355)
0925 14:51:01.603866 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.603868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.603898 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:51:01.603957 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.603958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.603966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.604012 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.604132 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.604675 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.605370 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.605546 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.605598 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.605715 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.605738 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.605770 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.605823 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.605837 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:51:01.606774 8343 raft_consensus.cc:1261] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.606781 8063 raft_consensus.cc:1261] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Refusing update from remote peer 4d89a18fd88f49a592b99c31f96e3791: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.607004 8423 consensus_queue.cc:1035] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.607127 8465 consensus_queue.cc:1035] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.609969 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.610072 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.610731 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Sending DeleteTablet for 3 replicas of tablet 0f057673a84c4127960531c3d5f1fdf2
I20260925 14:51:01.610965 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260925 14:51:01.610975 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57994
I20260925 14:51:01.611069 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f057673a84c4127960531c3d5f1fdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60420
I20260925 14:51:01.611166 8440 tablet_replica.cc:330] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: stopping tablet replica
I20260925 14:51:01.611243 8440 raft_consensus.cc:2229] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.611299 8440 raft_consensus.cc:2258] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.611241 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:01.611373 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:01.611373 8441 tablet_replica.cc:330] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: stopping tablet replica
I20260925 14:51:01.611429 8441 raft_consensus.cc:2229] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.611497 8441 raft_consensus.cc:2258] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.611567 8439 tablet_replica.cc:330] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: stopping tablet replica
I20260925 14:51:01.611620 8439 raft_consensus.cc:2229] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:01.611665 8440 ts_tablet_manager.cc:1905] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.611688 8439 raft_consensus.cc:2258] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.612002 8439 ts_tablet_manager.cc:1905] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.612203 8441 ts_tablet_manager.cc:1905] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.612493 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 606b70d5028746a2a4281a5756e868aa (DEFAULT_TABLE table=test-table [id=a465613d908b4724983b35690622ab9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.612589 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606b70d5028746a2a4281a5756e868aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.613263 8440 ts_tablet_manager.cc:1918] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.613328 8440 log.cc:1196] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/0f057673a84c4127960531c3d5f1fdf2
I20260925 14:51:01.613520 8439 ts_tablet_manager.cc:1918] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.613550 8440 ts_tablet_manager.cc:1939] T 0f057673a84c4127960531c3d5f1fdf2 P 06f4c3165b5740388f2b11d4ddded953: Deleting consensus metadata
I20260925 14:51:01.613574 8439 log.cc:1196] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/0f057673a84c4127960531c3d5f1fdf2
I20260925 14:51:01.613616 8441 ts_tablet_manager.cc:1918] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:01.613662 8441 log.cc:1196] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/0f057673a84c4127960531c3d5f1fdf2
I20260925 14:51:01.613802 8439 ts_tablet_manager.cc:1939] T 0f057673a84c4127960531c3d5f1fdf2 P 4d89a18fd88f49a592b99c31f96e3791: Deleting consensus metadata
I20260925 14:51:01.613816 8421 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Bootstrap starting.
I20260925 14:51:01.613866 8441 ts_tablet_manager.cc:1939] T 0f057673a84c4127960531c3d5f1fdf2 P f7d12dcbacbb4c418f740f17b7fc6671: Deleting consensus metadata
I20260925 14:51:01.614128 8421 tablet_bootstrap.cc:654] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.614516 7892 catalog_manager.cc:4841] TS 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287): tablet 0f057673a84c4127960531c3d5f1fdf2 (table test-table [id=0e4b39212273487da25d68c3896241ea]) successfully deleted
I20260925 14:51:01.614591 7893 catalog_manager.cc:4841] TS 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355): tablet 0f057673a84c4127960531c3d5f1fdf2 (table test-table [id=0e4b39212273487da25d68c3896241ea]) successfully deleted
I20260925 14:51:01.614737 8421 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: No bootstrap required, opened a new log
I20260925 14:51:01.614789 7893 catalog_manager.cc:4841] TS f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125): tablet 0f057673a84c4127960531c3d5f1fdf2 (table test-table [id=0e4b39212273487da25d68c3896241ea]) successfully deleted
I20260925 14:51:01.614797 8421 ts_tablet_manager.cc:1397] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:01.614861 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 606b70d5028746a2a4281a5756e868aa (DEFAULT_TABLE table=test-table [id=a465613d908b4724983b35690622ab9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.614904 8421 raft_consensus.cc:348] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.614954 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606b70d5028746a2a4281a5756e868aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.614979 8421 raft_consensus.cc:374] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.615005 8421 raft_consensus.cc:729] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4c3165b5740388f2b11d4ddded953, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.615049 8421 consensus_queue.cc:260] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.615137 8421 ts_tablet_manager.cc:1428] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.615260 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 606b70d5028746a2a4281a5756e868aa (DEFAULT_TABLE table=test-table [id=a465613d908b4724983b35690622ab9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:01.615355 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 606b70d5028746a2a4281a5756e868aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:01.616088 8417 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: Bootstrap starting.
I20260925 14:51:01.616528 8418 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Bootstrap starting.
I20260925 14:51:01.616755 8417 tablet_bootstrap.cc:654] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.616824 8418 tablet_bootstrap.cc:654] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:01.617357 8417 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: No bootstrap required, opened a new log
I20260925 14:51:01.617386 8418 tablet_bootstrap.cc:492] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: No bootstrap required, opened a new log
I20260925 14:51:01.617410 8417 ts_tablet_manager.cc:1397] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.617424 8418 ts_tablet_manager.cc:1397] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:01.617519 8418 raft_consensus.cc:348] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.617591 8418 raft_consensus.cc:374] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.617614 8418 raft_consensus.cc:729] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89a18fd88f49a592b99c31f96e3791, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.617570 8417 raft_consensus.cc:348] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.617633 8417 raft_consensus.cc:374] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:01.617663 8417 raft_consensus.cc:729] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Initialized, Role: FOLLOWER
I20260925 14:51:01.617654 8418 consensus_queue.cc:260] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.617710 8417 consensus_queue.cc:260] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.617743 8418 ts_tablet_manager.cc:1428] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:01.617815 8417 ts_tablet_manager.cc:1428] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:01.645329 8424 raft_consensus.cc:482] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:01.645416 8424 raft_consensus.cc:504] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.645812 8424 leader_election.cc:288] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.648746 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.646036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39909 (local address 127.0.98.3:34287)
0925 14:51:01.646090 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.646092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.646190 (+ 98us) server_negotiation.cc:407] Connection header received
0925 14:51:01.646219 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.646220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.646230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.646287 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.646503 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.646986 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648024 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.648391 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648443 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.648452 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.648630 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.648673 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:51:01.648765 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.645976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34287 (local address 127.0.98.1:39909)
0925 14:51:01.646082 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.646094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.646162 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.646309 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.646310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.646312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.646435 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.646437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.647081 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.647082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.647939 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.647941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.648486 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.648560 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.648564 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.648606 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.648662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260925 14:51:01.648772 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.646095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48323 (local address 127.0.98.2:34355)
0925 14:51:01.646140 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.646142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.646155 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:01.646260 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.646261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.646286 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.646366 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:01.646561 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.647014 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648063 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.648405 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648493 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.648504 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.648667 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.648701 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:51:01.648790 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.645985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.98.1:48323)
0925 14:51:01.646111 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.646120 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.646170 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.646390 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.646391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.646393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:01.646485 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.646487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.647126 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.647127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.647952 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.647955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.648505 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.648506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.648575 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.648578 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.648659 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.648694 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":52,"thread_start_us":56,"threads_started":1}
I20260925 14:51:01.649055 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606b70d5028746a2a4281a5756e868aa" candidate_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89a18fd88f49a592b99c31f96e3791" is_pre_election: true
I20260925 14:51:01.649075 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606b70d5028746a2a4281a5756e868aa" candidate_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953" is_pre_election: true
I20260925 14:51:01.649140 8203 raft_consensus.cc:2454] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d12dcbacbb4c418f740f17b7fc6671 in term 0.
I20260925 14:51:01.649171 8343 raft_consensus.cc:2454] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7d12dcbacbb4c418f740f17b7fc6671 in term 0.
I20260925 14:51:01.649323 7997 leader_election.cc:302] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89a18fd88f49a592b99c31f96e3791, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.649463 8424 raft_consensus.cc:2781] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:01.649501 8424 raft_consensus.cc:482] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:01.649516 8424 raft_consensus.cc:3037] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.650197 8424 raft_consensus.cc:504] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.650357 8424 leader_election.cc:288] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [CANDIDATE]: Term 1 election: Requested vote from peers 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355), 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287)
I20260925 14:51:01.650514 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606b70d5028746a2a4281a5756e868aa" candidate_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4c3165b5740388f2b11d4ddded953"
I20260925 14:51:01.650504 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "606b70d5028746a2a4281a5756e868aa" candidate_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89a18fd88f49a592b99c31f96e3791"
I20260925 14:51:01.650575 8203 raft_consensus.cc:3037] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.650593 8343 raft_consensus.cc:3037] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:01.651151 8343 raft_consensus.cc:2454] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d12dcbacbb4c418f740f17b7fc6671 in term 1.
I20260925 14:51:01.651268 8203 raft_consensus.cc:2454] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7d12dcbacbb4c418f740f17b7fc6671 in term 1.
I20260925 14:51:01.651394 7996 leader_election.cc:302] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4c3165b5740388f2b11d4ddded953, f7d12dcbacbb4c418f740f17b7fc6671; no voters:
I20260925 14:51:01.651477 8424 raft_consensus.cc:2781] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:01.651628 8424 raft_consensus.cc:686] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 LEADER]: Becoming Leader. State: Replica: f7d12dcbacbb4c418f740f17b7fc6671, State: Running, Role: LEADER
I20260925 14:51:01.651708 8424 consensus_queue.cc:237] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } }
I20260925 14:51:01.652460 7906 catalog_manager.cc:5495] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7d12dcbacbb4c418f740f17b7fc6671" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 } health_report { overall_health: UNKNOWN } } }
I20260925 14:51:01.667925 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.665560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.1:33125)
0925 14:51:01.665597 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.665598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.665630 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:51:01.665672 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.665673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.665689 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.665743 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.665861 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.666444 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.667241 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.667415 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.667469 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.667632 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.667734 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.667788 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.667858 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.667873 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:51:01.672344 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.670279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47795 (local address 127.0.98.2:34355)
0925 14:51:01.670373 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.670375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.670385 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:01.670409 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.670409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.670419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.670475 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.670549 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.670988 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.671745 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.671959 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.672011 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.672163 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.672186 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.672224 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.672274 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.672292 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:51:01.672370 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.670109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34355 (local address 127.0.98.1:47795)
0925 14:51:01.670281 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.670292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.670365 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.670476 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.670476 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.670477 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:51:01.670508 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.670509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.671056 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.671057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.671607 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.671609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.672040 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.672041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.672081 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.672093 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.672236 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.672238 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.672273 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.672307 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260925 14:51:01.674710 8343 raft_consensus.cc:1261] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Refusing update from remote peer f7d12dcbacbb4c418f740f17b7fc6671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.674755 8203 raft_consensus.cc:1261] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Refusing update from remote peer f7d12dcbacbb4c418f740f17b7fc6671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:01.674984 8424 consensus_queue.cc:1035] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d89a18fd88f49a592b99c31f96e3791" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.675309 8498 consensus_queue.cc:1035] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f4c3165b5740388f2b11d4ddded953" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:01.682179 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.680072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43429 (local address 127.0.98.1:33125)
0925 14:51:01.680112 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:01.680114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:01.680121 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:51:01.680141 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:01.680141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:01.680149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:01.680192 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:01.680288 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.680748 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.681484 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.681693 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.681764 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.681917 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.681948 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.682019 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:01.682080 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:51:01.682098 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:51:01.682188 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.679971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33125 (local address 127.0.98.2:43429)
0925 14:51:01.680005 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.680014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:01.680068 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:01.680203 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:01.680204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:01.680205 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:51:01.680243 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.680245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.680827 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.680828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.681385 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:01.681387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:01.681772 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:01.681773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:01.681824 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:01.681829 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:51:01.682020 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:51:01.682022 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:01.682075 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:51:01.682118 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 14:51:01.688750 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.688862 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:01.689499 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25fd120315a848a88d8be6c7ee44705e: Sending DeleteTablet for 3 replicas of tablet 606b70d5028746a2a4281a5756e868aa
I20260925 14:51:01.689730 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 606b70d5028746a2a4281a5756e868aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260925 14:51:01.689754 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 606b70d5028746a2a4281a5756e868aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57994
I20260925 14:51:01.689819 8440 tablet_replica.cc:330] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: stopping tablet replica
I20260925 14:51:01.689841 8441 tablet_replica.cc:330] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671: stopping tablet replica
I20260925 14:51:01.689860 8440 raft_consensus.cc:2229] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.689889 8440 raft_consensus.cc:2258] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.689886 8441 raft_consensus.cc:2229] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:01.689968 8441 raft_consensus.cc:2258] T 606b70d5028746a2a4281a5756e868aa P f7d12dcbacbb4c418f740f17b7fc6671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.690174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7978
I20260925 14:51:01.690277 8440 ts_tablet_manager.cc:1905] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.691918 8440 ts_tablet_manager.cc:1918] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:51:01.691979 8440 log.cc:1196] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/606b70d5028746a2a4281a5756e868aa
I20260925 14:51:01.692207 8440 ts_tablet_manager.cc:1939] T 606b70d5028746a2a4281a5756e868aa P 06f4c3165b5740388f2b11d4ddded953: Deleting consensus metadata
I20260925 14:51:01.692427 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 606b70d5028746a2a4281a5756e868aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60420
I20260925 14:51:01.692517 8439 tablet_replica.cc:330] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: stopping tablet replica
I20260925 14:51:01.692535 7892 catalog_manager.cc:4841] TS 06f4c3165b5740388f2b11d4ddded953 (127.0.98.3:34287): tablet 606b70d5028746a2a4281a5756e868aa (table test-table [id=a465613d908b4724983b35690622ab9d]) successfully deleted
I20260925 14:51:01.692564 8439 raft_consensus.cc:2229] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:01.692600 8439 raft_consensus.cc:2258] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:01.692889 8439 ts_tablet_manager.cc:1905] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:01.694118 8439 ts_tablet_manager.cc:1918] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:51:01.694166 8439 log.cc:1196] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/606b70d5028746a2a4281a5756e868aa
I20260925 14:51:01.694404 8439 ts_tablet_manager.cc:1939] T 606b70d5028746a2a4281a5756e868aa P 4d89a18fd88f49a592b99c31f96e3791: Deleting consensus metadata
I20260925 14:51:01.694732 7893 catalog_manager.cc:4841] TS 4d89a18fd88f49a592b99c31f96e3791 (127.0.98.2:34355): tablet 606b70d5028746a2a4281a5756e868aa (table test-table [id=a465613d908b4724983b35690622ab9d]) successfully deleted
W20260925 14:51:01.696215 7893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33125 (error 108)
I20260925 14:51:01.696509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8118
I20260925 14:51:01.697227 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:01.696605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33125 (local address 127.0.0.1:46312)
0925 14:51:01.697114 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:01.697195 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":39,"threads_started":1}
W20260925 14:51:01.698410 7893 catalog_manager.cc:4584] TS f7d12dcbacbb4c418f740f17b7fc6671 (127.0.98.1:33125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 606b70d5028746a2a4281a5756e868aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33125: connect: Connection refused (error 111)
I20260925 14:51:01.703028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8258
I20260925 14:51:01.709246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 7874
2026-09-25T14:51:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2564 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[8511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[8511](info): set up 2 sockets
Sep 25 14:51:01 dist-test-slave-gpc1 krb5kdc[8511](info): commencing operation
krb5kdc: starting...
W20260925 14:51:02.758900 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:02 dist-test-slave-gpc1 krb5kdc[8511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:02.875465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36301
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:02.959358 8524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:02.959560 8524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:02.959583 8524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:02.960678 8524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:02.960713 8524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:02.960727 8524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:02.960739 8524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:02.960752 8524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:02.962185 8524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:02.962441 8524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:02.962628 8524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:02.967422 8532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:02.967427 8530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:02.967592 8524 server_base.cc:1034] running on GCE node
W20260925 14:51:02.967442 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:02.967896 8524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:02.968189 8524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:02.969372 8524 hybrid_clock.cc:648] HybridClock initialized: now 1790347862969372 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:51:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1262 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:03 dist-test-slave-gpc1 krb5kdc[8537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:03 dist-test-slave-gpc1 krb5kdc[8537](info): set up 2 sockets
Sep 25 14:51:03 dist-test-slave-gpc1 krb5kdc[8537](info): commencing operation
krb5kdc: starting...
W20260925 14:51:04.047178 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:04.163188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:04.242235 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:04.242446 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:04.242468 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:04.243654 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:04.243698 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:04.243712 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:04.243726 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:04.243737 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:04.245242 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:04.245483 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:04.245719 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:04.250586 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.250561 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.250552 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:04.250819 8550 server_base.cc:1034] running on GCE node
I20260925 14:51:04.251091 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:04.251389 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:04.252552 8550 hybrid_clock.cc:648] HybridClock initialized: now 1790347864252539 us; error 47 us; skew 500 ppm
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:04.271049 8550 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:04.271569 8550 webserver.cc:460] Webserver started at http://127.0.98.62:33861/ using document root <none> and password file <none>
I20260925 14:51:04.271835 8550 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:04.271905 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:04.272051 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:04.273159 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "cadd007fc4a64852aa9346ae32fe357c"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "7081d9d04e1e94fa1d0d0ff1cc2738c7"
server_key_iv: "425d65738d2f0e851bb47300cadeedc2"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.273526 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "cadd007fc4a64852aa9346ae32fe357c"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "7081d9d04e1e94fa1d0d0ff1cc2738c7"
server_key_iv: "425d65738d2f0e851bb47300cadeedc2"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.274945 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:04.275635 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.275796 8550 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:04.275902 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "cadd007fc4a64852aa9346ae32fe357c"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "7081d9d04e1e94fa1d0d0ff1cc2738c7"
server_key_iv: "425d65738d2f0e851bb47300cadeedc2"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.275976 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:04.289386 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:04.291369 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:04.291531 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:04.295531 8550 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33581
I20260925 14:51:04.295579 8617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33581 every 8 connection(s)
I20260925 14:51:04.296088 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:04.296732 8618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:04.297753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8550
I20260925 14:51:04.297871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:04.298050 392 external_mini_cluster.cc:1428] Setting key 5aabf3fa6434bed0372725dbe60d12ed
I20260925 14:51:04.299314 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Bootstrap starting.
I20260925 14:51:04.300093 8618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:04.300395 8618 log.cc:824] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:04.301051 8618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: No bootstrap required, opened a new log
I20260925 14:51:04.303426 8618 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:04.303606 8618 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:04.303642 8618 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadd007fc4a64852aa9346ae32fe357c, State: Initialized, Role: FOLLOWER
I20260925 14:51:04.303781 8618 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:04.303867 8618 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:04.303905 8618 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:04.303960 8618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:04.304709 8618 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:04.304793 8618 leader_election.cc:302] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadd007fc4a64852aa9346ae32fe357c; no voters:
I20260925 14:51:04.304993 8618 leader_election.cc:288] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:04.305017 8623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Leader election won for term 1
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:04.305117 8623 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 LEADER]: Becoming Leader. State: Replica: cadd007fc4a64852aa9346ae32fe357c, State: Running, Role: LEADER
I20260925 14:51:04.305186 8623 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:04.305248 8618 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:04.305505 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadd007fc4a64852aa9346ae32fe357c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260925 14:51:04.305657 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:04.305848 8623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadd007fc4a64852aa9346ae32fe357c. Latest consensus state: current_term: 1 leader_uuid: "cadd007fc4a64852aa9346ae32fe357c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260925 14:51:04.305926 8623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:04.306290 8630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:04.306794 8630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:04.308548 8630 catalog_manager.cc:1347] Generated new cluster ID: 71fb21212bf24825ac4e1749a732388e
I20260925 14:51:04.308641 8630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:04.309415 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.298579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.98.62:33581)
0925 14:51:04.298771 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:04.298791 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:04.298816 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:51:04.299227 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:04.299233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:04.299280 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:04.299394 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:04.300236 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.301289 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.302412 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.302820 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.302904 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.305694 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:04.305697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:04.305698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:04.305711 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:04.307772 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.308157 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.308159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.308160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.308264 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.308513 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.308514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.308514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.308810 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:04.308990 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:04.309101 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:51:04.309253 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260925 14:51:04.316711 8630 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:04.317395 8630 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:04.322518 8630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Generated new TSK 0
I20260925 14:51:04.322693 8630 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:51:04.354122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:04.450162 8644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:04.450428 8644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:04.450474 8644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:04.451639 8644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:04.451678 8644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:04.451709 8644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:04.453204 8644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:04.453442 8644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:04.453679 8644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:04.458618 8649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.458595 8652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.458596 8650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:04.459052 8644 server_base.cc:1034] running on GCE node
I20260925 14:51:04.459276 8644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:04.459506 8644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:04.460669 8644 hybrid_clock.cc:648] HybridClock initialized: now 1790347864460650 us; error 32 us; skew 500 ppm
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:04.463353 8644 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:04.463831 8644 webserver.cc:460] Webserver started at http://127.0.98.1:33429/ using document root <none> and password file <none>
I20260925 14:51:04.464078 8644 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:04.464143 8644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:04.464285 8644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:04.465338 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "a59bf5101d6341c3ae722f8a7abafddf"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "5b277bbe09306a50147263f10c78f262"
server_key_iv: "69e50e0046c019db7eec9584639a7484"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.465685 8644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a59bf5101d6341c3ae722f8a7abafddf"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "5b277bbe09306a50147263f10c78f262"
server_key_iv: "69e50e0046c019db7eec9584639a7484"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.467149 8644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:04.467885 8659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.468079 8644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:04.468181 8644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "a59bf5101d6341c3ae722f8a7abafddf"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "5b277bbe09306a50147263f10c78f262"
server_key_iv: "69e50e0046c019db7eec9584639a7484"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.468253 8644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:04.496284 8644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:04.498281 8644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:04.498459 8644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:04.498713 8644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:04.499012 8644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:04.499061 8644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.499104 8644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:04.499135 8644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.505079 8644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39469
I20260925 14:51:04.505149 8772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39469 every 8 connection(s)
I20260925 14:51:04.505476 8644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:04.509784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8644
I20260925 14:51:04.509902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:04.510100 392 external_mini_cluster.cc:1428] Setting key 710d5194231a407a3e5849db2652d848
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:04.516448 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.506294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51843 (local address 127.0.98.62:33581)
0925 14:51:04.506336 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:04.506357 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:04.506604 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:51:04.507615 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:04.507616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:04.507628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:04.507695 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:04.508304 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.508868 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.510042 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.510381 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.510454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.513182 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:04.513183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:04.513184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:04.513191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:04.514848 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.515398 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.515399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.515399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.515457 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.515754 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.515755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.515755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.515866 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:04.515918 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:04.516296 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 14:51:04.516381 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:51:04.516723 8775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.506280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:51843)
0925 14:51:04.506475 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:04.506511 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:04.507509 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:04.507778 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:04.507781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:04.508029 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:04.508222 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.508226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.508946 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.508946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.509914 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.509917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.510462 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.510463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.510615 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.511237 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:04.511246 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:04.513021 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:04.514931 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.514939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.514942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.515320 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.515524 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.515526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.515526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.515688 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.515930 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:04.515936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:04.516140 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 14:51:04.516275 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:51:04.516461 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":127,"threads_started":1}
I20260925 14:51:04.517445 8773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260925 14:51:04.517573 8773 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:04.517796 8773 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260925 14:51:04.518419 8582 ts_manager.cc:194] Registered new tserver with Master: a59bf5101d6341c3ae722f8a7abafddf (127.0.98.1:39469)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:51:04.519166 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:51:04.537135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:04.618058 8778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:04.618314 8778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:04.618353 8778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:04.619544 8778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:04.619609 8778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:04.619649 8778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:04.621122 8778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:04.621369 8778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:04.621580 8778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:04.626618 8786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.626694 8784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:04.626849 8778 server_base.cc:1034] running on GCE node
W20260925 14:51:04.626617 8783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:04.627126 8778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:04.627388 8778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:04.628528 8778 hybrid_clock.cc:648] HybridClock initialized: now 1790347864628515 us; error 36 us; skew 500 ppm
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:04.631021 8778 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:04.631497 8778 webserver.cc:460] Webserver started at http://127.0.98.2:44847/ using document root <none> and password file <none>
I20260925 14:51:04.631671 8778 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:04.631711 8778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:04.631799 8778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:04.632784 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e9f422228d04e0e8ac44d2932da4e64"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "4ca6b2165bf2f5c437f2a4a46c8f5ed6"
server_key_iv: "e477a58c7afa35783143615acf8f9a43"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.633123 8778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e9f422228d04e0e8ac44d2932da4e64"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "4ca6b2165bf2f5c437f2a4a46c8f5ed6"
server_key_iv: "e477a58c7afa35783143615acf8f9a43"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.634415 8778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:04.635098 8793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.635280 8778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:04.635375 8778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "1e9f422228d04e0e8ac44d2932da4e64"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "4ca6b2165bf2f5c437f2a4a46c8f5ed6"
server_key_iv: "e477a58c7afa35783143615acf8f9a43"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.635443 8778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:04.655395 8778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:04.657290 8778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:04.657444 8778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:04.657673 8778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:04.657974 8778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:04.658021 8778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.658063 8778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:04.658100 8778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.663908 8778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41623
I20260925 14:51:04.663950 8906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41623 every 8 connection(s)
I20260925 14:51:04.664297 8778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:04.671903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8778
I20260925 14:51:04.672034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:04.672240 392 external_mini_cluster.cc:1428] Setting key 668c983c71d8dfee1dd88e8e46a574fc
I20260925 14:51:04.673847 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.664966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34729 (local address 127.0.98.62:33581)
0925 14:51:04.665006 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:04.665008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:04.665229 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:51:04.666197 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:04.666199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:04.666212 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:04.666259 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:04.666932 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.667428 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.668418 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.668611 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.668684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.670690 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:04.670691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:04.670691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:04.670698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:04.672380 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.672858 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.672859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.672860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.672924 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.673173 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.673174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.673175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.673319 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:04.673390 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:04.673621 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 14:51:04.673706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:51:04.673869 8909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.664946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:34729)
0925 14:51:04.665096 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:04.665130 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:04.666080 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:04.666369 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:04.666373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:04.666634 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:04.666830 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.666834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.667515 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.667516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.668331 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.668333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.668691 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.668692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.668814 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.669275 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:04.669283 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:04.670606 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:04.672461 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.672467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.672469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.672775 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.672995 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.672996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.672996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.673105 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.673384 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:04.673388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:04.673499 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:51:04.673581 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:51:04.673713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260925 14:51:04.674734 8907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260925 14:51:04.674836 8907 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:04.675005 8907 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260925 14:51:04.675374 8582 ts_manager.cc:194] Registered new tserver with Master: 1e9f422228d04e0e8ac44d2932da4e64 (127.0.98.2:41623)
I20260925 14:51:04.675757 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34729
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:51:04.695350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:04.774923 8912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:04.775125 8912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:04.775152 8912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:04.776270 8912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:04.776307 8912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:04.776335 8912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:04.777786 8912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:04.777982 8912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:04.778157 8912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:04.783198 8917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.783267 8920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:04.783209 8918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:04.783459 8912 server_base.cc:1034] running on GCE node
I20260925 14:51:04.783705 8912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:04.783941 8912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:04.785094 8912 hybrid_clock.cc:648] HybridClock initialized: now 1790347864785086 us; error 37 us; skew 500 ppm
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:04.787563 8912 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:04.787971 8912 webserver.cc:460] Webserver started at http://127.0.98.3:41869/ using document root <none> and password file <none>
I20260925 14:51:04.788199 8912 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:04.788264 8912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:04.788400 8912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:04.789463 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f64c2d4cd7441cdaf736e4c17b0fbc7"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "11d5cd3c2bf6cba39a8ab047ff1a39b8"
server_key_iv: "8bf9b6924807d25ef4b91fb9da1a6644"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.789808 8912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f64c2d4cd7441cdaf736e4c17b0fbc7"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "11d5cd3c2bf6cba39a8ab047ff1a39b8"
server_key_iv: "8bf9b6924807d25ef4b91fb9da1a6644"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.791258 8912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:04.791971 8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.792146 8912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:04.792274 8912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "6f64c2d4cd7441cdaf736e4c17b0fbc7"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "11d5cd3c2bf6cba39a8ab047ff1a39b8"
server_key_iv: "8bf9b6924807d25ef4b91fb9da1a6644"
server_key_version: "encryptionkey@0"
I20260925 14:51:04.792346 8912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:04.820910 8912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:04.822919 8912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:04.823082 8912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:04.823316 8912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:04.823602 8912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:04.823649 8912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.823691 8912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:04.823736 8912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:04.829491 8912 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34389
I20260925 14:51:04.829558 9040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34389 every 8 connection(s)
I20260925 14:51:04.829875 8912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:51:04 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:04.838585 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.830629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58397 (local address 127.0.98.62:33581)
0925 14:51:04.830671 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:04.830672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:04.830882 (+ 210us) server_negotiation.cc:407] Connection header received
0925 14:51:04.831800 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:04.831802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:04.831812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:04.831860 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:04.832428 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.832915 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.833941 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.834131 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.834222 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.836237 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:04.836238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:04.836238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:04.836244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:04.837382 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.837727 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.837728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.837728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.837775 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:04.838018 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:04.838019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:04.838019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:04.838122 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:04.838174 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:04.838464 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 14:51:04.838523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:04.838845 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:04.830608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:58397)
0925 14:51:04.830779 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:04.830814 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:04.831718 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:04.831923 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:04.831927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:04.832176 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:04.832360 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.832363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.832991 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.832992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.833841 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:04.833842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:04.834207 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:04.834208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:04.834307 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:04.834867 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:04.834881 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:04.836151 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:04.837443 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.837448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.837450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.837667 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.837826 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:04.837826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:04.837827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:04.837936 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:04.838175 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:04.838179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:04.838285 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:51:04.838443 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 14:51:04.838569 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260925 14:51:04.839398 9041 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33581
I20260925 14:51:04.839497 9041 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:04.839676 9041 heartbeater.cc:511] Master 127.0.98.62:33581 requested a full tablet report, sending...
I20260925 14:51:04.839946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 8912
I20260925 14:51:04.840029 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:04.840113 8582 ts_manager.cc:194] Registered new tserver with Master: 6f64c2d4cd7441cdaf736e4c17b0fbc7 (127.0.98.3:34389)
I20260925 14:51:04.840240 392 external_mini_cluster.cc:1428] Setting key 3bffe71601dce189b0a09a6dd5301392
I20260925 14:51:04.840518 8582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58397
I20260925 14:51:04.841094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:51:04.909229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8644
I20260925 14:51:04.914259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8778
I20260925 14:51:04.918598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8912
I20260925 14:51:04.922698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 8550
I20260925 14:51:04.927738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--webserver_interface=127.0.98.62
--webserver_port=33861
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:05.007145 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:05.007359 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:05.007380 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:05.008561 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:05.008605 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:05.008620 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:05.008632 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:05.008680 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:05.010145 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:05.010419 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:05.010643 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:05.015344 9056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.015337 9058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.015337 9055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.015363 9050 server_base.cc:1034] running on GCE node
I20260925 14:51:05.015791 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:05.016073 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:05.017237 9050 hybrid_clock.cc:648] HybridClock initialized: now 1790347865017225 us; error 40 us; skew 500 ppm
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:05.019667 9050 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:05.020040 9050 webserver.cc:460] Webserver started at http://127.0.98.62:33861/ using document root <none> and password file <none>
I20260925 14:51:05.020212 9050 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:05.020263 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:05.021543 9050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:05.022217 9065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.022392 9050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:05.022516 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "cadd007fc4a64852aa9346ae32fe357c"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "7081d9d04e1e94fa1d0d0ff1cc2738c7"
server_key_iv: "425d65738d2f0e851bb47300cadeedc2"
server_key_version: "encryptionkey@0"
I20260925 14:51:05.022950 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:05.043584 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:05.045948 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:05.046098 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:05.049894 9050 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33581
I20260925 14:51:05.049955 9117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33581 every 8 connection(s)
I20260925 14:51:05.050529 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:05.051843 9118 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:51:05.052136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9050
I20260925 14:51:05.052467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39469
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33581
--webserver_port=33429
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:05.052825 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Bootstrap starting.
I20260925 14:51:05.061082 9118 log.cc:824] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:05.062819 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:05.063099 9118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Bootstrap complete.
I20260925 14:51:05.065286 9118 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:05.065551 9118 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadd007fc4a64852aa9346ae32fe357c, State: Initialized, Role: FOLLOWER
I20260925 14:51:05.065757 9118 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:05.065841 9118 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:05.065912 9118 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:05.065961 9118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:51:05.066987 9118 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:05.067101 9118 leader_election.cc:302] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadd007fc4a64852aa9346ae32fe357c; no voters:
I20260925 14:51:05.067289 9118 leader_election.cc:288] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:51:05.067323 9122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:51:05.067428 9122 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [term 2 LEADER]: Becoming Leader. State: Replica: cadd007fc4a64852aa9346ae32fe357c, State: Running, Role: LEADER
I20260925 14:51:05.067551 9118 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:05.067584 9122 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260925 14:51:05.067847 9122 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadd007fc4a64852aa9346ae32fe357c. Latest consensus state: current_term: 2 leader_uuid: "cadd007fc4a64852aa9346ae32fe357c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260925 14:51:05.067916 9122 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:05.068051 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cadd007fc4a64852aa9346ae32fe357c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd007fc4a64852aa9346ae32fe357c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260925 14:51:05.068125 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:05.069396 9138 catalog_manager.cc:1259] Loaded cluster ID: 71fb21212bf24825ac4e1749a732388e
I20260925 14:51:05.069443 9138 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: loading cluster ID for follower catalog manager: success
I20260925 14:51:05.070600 9138 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: acquiring CA information for follower catalog manager: success
I20260925 14:51:05.070914 9138 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:51:05.070988 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:05.071111 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:05.071174 9127 catalog_manager.cc:1259] Loaded cluster ID: 71fb21212bf24825ac4e1749a732388e
I20260925 14:51:05.071221 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:05.071388 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:05.071467 9127 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cadd007fc4a64852aa9346ae32fe357c: Loaded TSK: 0
I20260925 14:51:05.071609 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:51:05.133014 9120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:05.133260 9120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:05.133298 9120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:05.134511 9120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:05.134574 9120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:05.134616 9120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:05.136046 9120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33429
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:05.136317 9120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:05.136541 9120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:05.142405 9144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.142473 9147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.142602 9120 server_base.cc:1034] running on GCE node
W20260925 14:51:05.142408 9145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.142917 9120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:05.143180 9120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:05.144366 9120 hybrid_clock.cc:648] HybridClock initialized: now 1790347865144351 us; error 29 us; skew 500 ppm
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:05.147159 9120 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:51:05.147740 9120 webserver.cc:460] Webserver started at http://127.0.98.1:33429/ using document root <none> and password file <none>
I20260925 14:51:05.147996 9120 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:05.148072 9120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:05.149811 9120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:51:05.150484 9154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.150684 9120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:05.150779 9120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "a59bf5101d6341c3ae722f8a7abafddf"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "5b277bbe09306a50147263f10c78f262"
server_key_iv: "69e50e0046c019db7eec9584639a7484"
server_key_version: "encryptionkey@0"
I20260925 14:51:05.151233 9120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:05.189311 9120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:05.191885 9120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:05.192036 9120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:05.192288 9120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:05.192626 9120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:05.192667 9120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.192700 9120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:05.192725 9120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.199734 9120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39469
I20260925 14:51:05.199775 9267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39469 every 8 connection(s)
I20260925 14:51:05.200157 9120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:05.207792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9120
I20260925 14:51:05.208151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41623
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33581
--webserver_port=44847
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.210665 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.200944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:34509)
0925 14:51:05.201117 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.201160 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.201632 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.202919 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.202923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.203500 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.203701 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.203705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.205307 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.205307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.206087 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.206089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.206884 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.206886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.207031 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.207642 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.207654 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.210464 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260925 14:51:05.210919 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.201009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34509 (local address 127.0.98.62:33581)
0925 14:51:05.202471 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.202482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.202528 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:51:05.202656 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.202662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.202716 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.202860 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.203810 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.205205 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.206164 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.206781 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.206878 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.210622 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34509: BlockingRecv error: recv got EOF from 127.0.98.1:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":94,"threads_started":1}
W20260925 14:51:05.211064 9268 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33581 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33581: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.215610 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.211328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:55687)
0925 14:51:05.211376 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.211387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.211429 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.211560 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.211561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.211673 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.211752 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.211754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.212433 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.212434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.212945 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.212946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.213308 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.213309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.213379 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.213860 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.213864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.215526 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:51:05.215663 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.211372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55687 (local address 127.0.98.62:33581)
0925 14:51:05.211406 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.211408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.211417 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:05.211474 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.211475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.211486 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.211543 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.211846 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.212343 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.213036 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.213230 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.213300 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.215600 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55687: BlockingRecv error: recv got EOF from 127.0.98.1:55687 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.219887 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.215875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:40033)
0925 14:51:05.215919 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.215927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.215954 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.216118 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.216119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.216239 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.216265 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.216266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.216874 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.216875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.217357 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.217359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.217708 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.217709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.217760 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.218211 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.218215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.219819 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:51:05.219954 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.215945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40033 (local address 127.0.98.62:33581)
0925 14:51:05.216000 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.216002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.216009 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:51:05.216030 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.216031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.216040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.216113 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.216341 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.216789 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.217448 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.217639 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.217708 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.219890 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40033: BlockingRecv error: recv got EOF from 127.0.98.1:40033 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 14:51:05.220023 9268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:51:05.289999 9272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:05.290226 9272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:05.290261 9272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:05.291471 9272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:05.291535 9272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:05.291579 9272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:05.293107 9272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44847
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:05.293339 9272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:05.293563 9272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:51:05.298446 9272 server_base.cc:1034] running on GCE node
W20260925 14:51:05.298419 9278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.298385 9280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.298384 9277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.298844 9272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:05.299058 9272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:05.300230 9272 hybrid_clock.cc:648] HybridClock initialized: now 1790347865300216 us; error 32 us; skew 500 ppm
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:05.302575 9272 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:51:05.302945 9272 webserver.cc:460] Webserver started at http://127.0.98.2:44847/ using document root <none> and password file <none>
I20260925 14:51:05.303133 9272 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:05.303216 9272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:05.304483 9272 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:05.305069 9287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.305332 9272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:05.305426 9272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "1e9f422228d04e0e8ac44d2932da4e64"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "4ca6b2165bf2f5c437f2a4a46c8f5ed6"
server_key_iv: "e477a58c7afa35783143615acf8f9a43"
server_key_version: "encryptionkey@0"
I20260925 14:51:05.305763 9272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:05.312397 9272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:05.314527 9272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:05.314673 9272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:05.314890 9272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:05.315162 9272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:05.315208 9272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.315249 9272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:05.315274 9272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.320889 9272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41623
I20260925 14:51:05.320945 9400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41623 every 8 connection(s)
I20260925 14:51:05.321291 9272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:05.323544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9272
I20260925 14:51:05.323858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34389
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33581
--webserver_port=41869
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.328962 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.321986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:40579)
0925 14:51:05.322166 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.322201 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.322731 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.322987 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.322996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.323526 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.323779 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.323784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.324552 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.324553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.325416 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.325419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.325865 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.325866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.325965 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.326556 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.326564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.328804 (+ 2240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":114,"threads_started":1}
W20260925 14:51:05.329030 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.322013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40579 (local address 127.0.98.62:33581)
0925 14:51:05.322067 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.322069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.322279 (+ 210us) server_negotiation.cc:407] Connection header received
0925 14:51:05.322842 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.322843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.322860 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.322906 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.323867 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.324459 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.325505 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.325704 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.325791 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.328944 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40579: BlockingRecv error: recv got EOF from 127.0.98.2:40579 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 14:51:05.329340 9401 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33581 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33581: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.334482 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.329536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:60709)
0925 14:51:05.329642 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.329650 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.329689 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.329811 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.329812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.329913 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.329950 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.329951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.330901 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.330901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.331396 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.331398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.331911 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.331912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.331960 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.332440 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.332445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.334418 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 14:51:05.334632 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.329592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60709 (local address 127.0.98.62:33581)
0925 14:51:05.329654 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.329656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.329682 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:51:05.329732 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.329733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.329743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.329807 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.330014 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.330812 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.331474 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.331818 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.331911 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.334494 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60709: BlockingRecv error: recv got EOF from 127.0.98.2:60709 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.338685 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.334807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:48833)
0925 14:51:05.334850 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.334861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.334899 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.335002 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.335003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.335137 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.335160 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.335161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.335742 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.335743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.336229 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.336230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.336543 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.336544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.336593 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.336962 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.336966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.338608 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:51:05.338733 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.334872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48833 (local address 127.0.98.62:33581)
0925 14:51:05.334911 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.334912 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.334921 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:05.334943 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.334944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.334953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.335001 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.335234 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.335670 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.336304 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.336480 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.336545 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.338671 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48833: BlockingRecv error: recv got EOF from 127.0.98.2:48833 (error 108)
Metrics: {"server-negotiator.queue_time_us":25,"thread_start_us":54,"threads_started":1}
W20260925 14:51:05.338790 9401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:51:05.407604 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:05.407816 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:05.407855 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:05.409193 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:05.409252 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:05.409296 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:05.410743 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41869
--tserver_master_addrs=127.0.98.62:33581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:05.410995 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:05.411221 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:05.416191 9411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:05.416294 9410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.416270 9404 server_base.cc:1034] running on GCE node
W20260925 14:51:05.416198 9413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:05.416586 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:05.416790 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:05.417933 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790347865417921 us; error 28 us; skew 500 ppm
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347865, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:05.420471 9404 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:51:05.420922 9404 webserver.cc:460] Webserver started at http://127.0.98.3:41869/ using document root <none> and password file <none>
I20260925 14:51:05.421118 9404 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:05.421168 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:05.422535 9404 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:05.423017 9420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.423173 9404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.423255 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "6f64c2d4cd7441cdaf736e4c17b0fbc7"
format_stamp: "Formatted at 2026-09-25 14:51:04 on dist-test-slave-gpc1"
server_key: "11d5cd3c2bf6cba39a8ab047ff1a39b8"
server_key_iv: "8bf9b6924807d25ef4b91fb9da1a6644"
server_key_version: "encryptionkey@0"
I20260925 14:51:05.423604 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:05.434610 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:05.436801 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:05.436951 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:05.437171 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:05.437460 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:05.437491 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.437529 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:05.437556 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:05.443347 9404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34389
I20260925 14:51:05.443395 9533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34389 every 8 connection(s)
I20260925 14:51:05.443719 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:05.448700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9404
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:51:05.451303 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.444354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42209 (local address 127.0.98.62:33581)
0925 14:51:05.444416 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.444418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.444643 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:51:05.445196 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.445197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.445210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.445262 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.446213 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.446761 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.447751 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.447949 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.448020 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.451258 (+ 3238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42209: BlockingRecv error: recv got EOF from 127.0.98.3:42209 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:51:05.451416 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.444357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:42209)
0925 14:51:05.444505 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.444556 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.445055 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.445342 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.445351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.445906 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.446098 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.446103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.446847 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.446848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.447652 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.447654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.448023 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.448024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.448147 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.448641 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.448651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.451188 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260925 14:51:05.451617 9534 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33581 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:33581: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.456429 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.451814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:46165)
0925 14:51:05.451948 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.451958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.451998 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.452123 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.452124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.452224 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.452263 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.452264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.452998 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.452999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.453793 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.453795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.454223 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.454224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.454272 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.454689 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.454694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.456352 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260925 14:51:05.456494 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.451859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46165 (local address 127.0.98.62:33581)
0925 14:51:05.451893 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.451895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.451989 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:51:05.452051 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.452051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.452060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.452120 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.452346 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.452927 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.453860 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.454127 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.454218 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.456444 (+ 2226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46165: BlockingRecv error: recv got EOF from 127.0.98.3:46165 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:51:05.457232 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.449247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.0.98.62:33581)
0925 14:51:05.449331 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.449334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.449342 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:05.449419 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.449420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.449432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.449498 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.450234 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.450820 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.451964 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.452245 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.452293 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.453642 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:05.453645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:05.453645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:05.453654 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:05.455732 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:05.456080 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:05.456083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:05.456084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:05.456197 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:05.456396 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:05.456397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:05.456397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:05.456671 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:05.456816 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:05.456999 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 14:51:05.457117 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:05 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:05.460513 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:05.456786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:50125)
0925 14:51:05.456816 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:05.456827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:05.456862 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:05.457006 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:05.457007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:05.457140 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:05.457168 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.457170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.457905 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.457906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.458527 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:05.458529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.458826 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.458827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:05.458860 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.459153 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:05.459156 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:05.460475 (+ 1319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260925 14:51:05.460621 9271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:05.456869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50125 (local address 127.0.98.62:33581)
0925 14:51:05.456907 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:05.456909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:05.456916 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:51:05.456934 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:05.456935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:05.456942 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:05.456994 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:05.457230 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.457836 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.458597 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:05.458773 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:05.458818 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:05.460555 (+ 1737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50125: BlockingRecv error: recv got EOF from 127.0.98.3:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:51:05.460644 9534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:06.224978 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:06.220513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:56255)
0925 14:51:06.220636 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:06.220648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:06.220707 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:06.220846 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:06.220847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:06.220995 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:06.221040 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.221041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.221685 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.221686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.222302 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.222304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.222630 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.222631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.222671 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.223140 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:06.223145 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:06.224912 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
W20260925 14:51:06.225128 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:06.220562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56255 (local address 127.0.98.62:33581)
0925 14:51:06.220696 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:06.220698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:06.220709 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:06.220765 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:06.220765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:06.220774 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:06.220835 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:06.221098 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.221611 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.222395 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.222572 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.222621 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.224999 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56255: BlockingRecv error: recv got EOF from 127.0.98.1:56255 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:06.344316 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:06.339207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:38539)
0925 14:51:06.339334 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:06.339350 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:06.339436 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:06.339563 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:06.339564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:06.339720 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:06.339768 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.339770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.340457 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.340459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.341147 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.341151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.341652 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.341653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.341738 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.342336 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:06.342378 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:06.344243 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260925 14:51:06.344465 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:06.339255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38539 (local address 127.0.98.62:33581)
0925 14:51:06.339328 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:06.339329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:06.339426 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:51:06.339486 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:06.339486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:06.339497 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:06.339548 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:06.339824 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.340350 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.341241 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.341553 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.341625 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.344367 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38539: BlockingRecv error: recv got EOF from 127.0.98.2:38539 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:06 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:06.465586 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:06.461021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:40123)
0925 14:51:06.461196 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:06.461207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:06.461265 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:06.461381 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:06.461382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:06.461552 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:06.461630 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.461631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.462250 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.462251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.462925 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:06.462927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.463237 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.463238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:06.463289 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.463753 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:06.463758 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:06.465509 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":68,"threads_started":1}
W20260925 14:51:06.465636 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:06.461083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40123 (local address 127.0.98.62:33581)
0925 14:51:06.461152 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:06.461154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:06.461262 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:51:06.461308 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:06.461309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:06.461320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:06.461367 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:06.461688 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.462187 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.462995 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:06.463178 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:06.463226 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:06.465589 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40123: BlockingRecv error: recv got EOF from 127.0.98.3:40123 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:07.230079 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:07.225467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:51143)
0925 14:51:07.225598 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:07.225610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:07.225706 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:07.225868 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:07.225869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:07.226054 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:07.226101 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.226103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.226839 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.226840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.227491 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.227494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.227842 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.227843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.227890 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.228356 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:07.228361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:07.230017 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
W20260925 14:51:07.230183 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:07.225509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51143 (local address 127.0.98.62:33581)
0925 14:51:07.225623 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:07.225625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:07.225729 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:51:07.225755 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:07.225756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:07.225768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:07.225838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:07.226161 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.226752 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.227567 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.227772 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.227826 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.230101 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51143: BlockingRecv error: recv got EOF from 127.0.98.1:51143 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":42,"threads_started":1}
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:07.348937 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:07.344754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:37047)
0925 14:51:07.344915 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:07.344925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:07.344974 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:07.345090 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:07.345090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:07.345244 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:07.345290 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.345291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.345892 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.345892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.346438 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.346440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.346774 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.346775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.346811 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.347244 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:07.347248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:07.348874 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260925 14:51:07.349028 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:07.344821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37047 (local address 127.0.98.62:33581)
0925 14:51:07.344872 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:07.344874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:07.344972 (+ 98us) server_negotiation.cc:407] Connection header received
0925 14:51:07.345016 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:07.345016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:07.345027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:07.345083 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:07.345339 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.345830 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.346511 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.346719 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.346767 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.348968 (+ 2201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37047: BlockingRecv error: recv got EOF from 127.0.98.2:37047 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:07 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:07.470873 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:07.466046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:56431)
0925 14:51:07.466191 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:07.466203 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:07.466273 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:07.466542 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:07.466543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:07.466717 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:07.466765 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.466767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.467445 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.467446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.468077 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:07.468079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.468545 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.468546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:07.468608 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.469039 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:07.469044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:07.470815 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
W20260925 14:51:07.470999 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:07.466193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56431 (local address 127.0.98.62:33581)
0925 14:51:07.466238 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:07.466240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:07.466253 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:07.466386 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:07.466387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:07.466404 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:07.466469 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:07.466839 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.467369 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.468158 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:07.468433 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:07.468512 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:07.470903 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56431: BlockingRecv error: recv got EOF from 127.0.98.3:56431 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:08.235129 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:08.230631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:35407)
0925 14:51:08.230775 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:08.230785 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:08.230839 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:08.230960 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:08.230961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:08.231120 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:08.231199 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.231200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.231841 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.231841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.232436 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.232439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.232913 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.232914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.232959 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.233370 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:08.233374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:08.235071 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260925 14:51:08.235270 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:08.230706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35407 (local address 127.0.98.62:33581)
0925 14:51:08.230845 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:08.230847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:08.230857 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:08.230880 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:08.230880 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:08.230889 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:08.230947 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:08.231262 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.231781 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.232525 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.232852 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.232904 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.235177 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35407: BlockingRecv error: recv got EOF from 127.0.98.1:35407 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:08.353829 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:08.349406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:33273)
0925 14:51:08.349531 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:08.349543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:08.349615 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:08.349780 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:08.349781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:08.349944 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:08.349992 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.349994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.350764 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.350765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.351291 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.351293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.351619 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.351620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.351661 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.352083 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:08.352088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:08.353774 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
W20260925 14:51:08.353919 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:08.349480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33273 (local address 127.0.98.62:33581)
0925 14:51:08.349548 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:08.349550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:08.349595 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:51:08.349670 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:08.349671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:08.349681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:08.349738 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:08.350072 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.350690 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.351359 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.351536 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.351601 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.353858 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33273: BlockingRecv error: recv got EOF from 127.0.98.2:33273 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:08 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:08.477370 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:08.471326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:59713)
0925 14:51:08.471479 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:08.471492 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:08.471573 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:08.471727 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:08.471727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:08.471948 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:08.472015 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.472018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.473032 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.473033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.473831 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:08.473833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.474286 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.474288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:08.474409 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.475041 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:08.475047 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:08.477295 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
W20260925 14:51:08.477453 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:08.471396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59713 (local address 127.0.98.62:33581)
0925 14:51:08.471446 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:08.471449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:08.471566 (+ 117us) server_negotiation.cc:407] Connection header received
0925 14:51:08.471634 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:08.471634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:08.471646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:08.471716 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:08.472088 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.472954 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.473926 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:08.474197 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:08.474266 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:08.477386 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59713: BlockingRecv error: recv got EOF from 127.0.98.3:59713 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:09.240164 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:09.235657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:47753)
0925 14:51:09.235799 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:09.235809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:09.235860 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:09.236029 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:09.236030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:09.236218 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:09.236266 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.236268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.236898 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.236899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.237409 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.237410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.237734 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.237735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.237772 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.238299 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:09.238304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:09.240104 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260925 14:51:09.240334 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:09.235744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47753 (local address 127.0.98.62:33581)
0925 14:51:09.235860 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:09.235862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:09.235872 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:09.235930 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:09.235931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:09.235945 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:09.236013 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:09.236330 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.236837 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.237476 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.237673 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.237727 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.240232 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47753: BlockingRecv error: recv got EOF from 127.0.98.1:47753 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:09.362305 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:09.354334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:46831)
0925 14:51:09.354688 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:09.354697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:09.354758 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:09.354898 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:09.354899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:09.355056 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:09.355102 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.355103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.355787 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.355788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.356297 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.356298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.356641 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.356642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.356685 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.360482 (+ 3797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:09.360487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:09.362229 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":110,"threads_started":1}
W20260925 14:51:09.362471 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:09.354454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46831 (local address 127.0.98.62:33581)
0925 14:51:09.354516 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:09.354518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:09.354768 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:51:09.354800 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:09.354801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:09.354816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:09.354887 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:09.355171 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.355717 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.356369 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.356560 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.356640 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.362411 (+ 5771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46831: BlockingRecv error: recv got EOF from 127.0.98.2:46831 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:09 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:09.482281 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:09.477964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:37491)
0925 14:51:09.478094 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:09.478105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:09.478165 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:09.478293 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:09.478294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:09.478504 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:09.478553 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.478555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.479191 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.479191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.479708 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:09.479709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.480038 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.480038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:09.480076 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.480528 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:09.480532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:09.482220 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
W20260925 14:51:09.482443 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:09.478048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37491 (local address 127.0.98.62:33581)
0925 14:51:09.478156 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:09.478158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:09.478169 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:09.478209 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:09.478210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:09.478221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:09.478280 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:09.478628 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.479124 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.479781 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:09.479972 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:09.480028 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:09.482324 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37491: BlockingRecv error: recv got EOF from 127.0.98.3:37491 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:10.245182 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:10.240685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:42611)
0925 14:51:10.240856 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:10.240867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:10.240923 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:10.241049 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:10.241050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:10.241225 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:10.241275 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.241276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.241898 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.241899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.242535 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.242538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.242838 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.242839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.242875 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.243312 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:10.243317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:10.245074 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
W20260925 14:51:10.245216 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:10.240751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42611 (local address 127.0.98.62:33581)
0925 14:51:10.240859 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:10.240861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:10.240924 (+ 63us) server_negotiation.cc:407] Connection header received
0925 14:51:10.240975 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:10.240976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:10.240985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:10.241034 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:10.241341 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.241830 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.242601 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.242782 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.242829 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.245149 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42611: BlockingRecv error: recv got EOF from 127.0.98.1:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:10.367473 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:10.362816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:32783)
0925 14:51:10.362953 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:10.362969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:10.363025 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:10.363147 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:10.363148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:10.363312 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:10.363361 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.363363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.363990 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.363991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.364522 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.364524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.364864 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.364866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.364918 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.365470 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:10.365474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:10.367365 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260925 14:51:10.367506 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:10.362928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32783 (local address 127.0.98.62:33581)
0925 14:51:10.362972 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:10.362974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:10.363014 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:51:10.363081 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:10.363082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:10.363091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:10.363132 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:10.363423 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.363927 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.364588 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.364776 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.364826 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.367443 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32783: BlockingRecv error: recv got EOF from 127.0.98.2:32783 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:10 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:10.487449 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:10.482767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:58283)
0925 14:51:10.482904 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:10.482917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:10.482998 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:10.483138 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:10.483138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:10.483315 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:10.483363 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.483365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.484011 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.484012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.484557 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:10.484558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.484975 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.484976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:10.485050 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.485593 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:10.485598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:10.487342 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260925 14:51:10.487480 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:10.482841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58283 (local address 127.0.98.62:33581)
0925 14:51:10.482907 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:10.482910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:10.482977 (+ 67us) server_negotiation.cc:407] Connection header received
0925 14:51:10.483060 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:10.483060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:10.483074 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:10.483131 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:10.483439 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.483929 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.484638 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:10.484889 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:10.484975 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:10.487417 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58283: BlockingRecv error: recv got EOF from 127.0.98.3:58283 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:11.250049 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:11.245640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:38821)
0925 14:51:11.245788 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:11.245798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:11.245846 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:11.245988 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:11.245989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:11.246165 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:11.246213 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.246215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.246837 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.246838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.247369 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.247370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.247686 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.247687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.247725 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.248212 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:11.248216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:11.249943 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
W20260925 14:51:11.250105 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:11.245696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38821 (local address 127.0.98.62:33581)
0925 14:51:11.245819 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:11.245821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:11.245834 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:11.245911 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:11.245911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:11.245921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:11.245971 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:11.246273 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.246779 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.247434 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.247625 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.247674 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.250021 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38821: BlockingRecv error: recv got EOF from 127.0.98.1:38821 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:11.372611 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:11.367932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:58051)
0925 14:51:11.368087 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:11.368096 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:11.368158 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:11.368281 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:11.368282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:11.368458 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:11.368540 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.368542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.369260 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.369261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.369776 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.369778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.370145 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.370145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.370181 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.370687 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:11.370691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:11.372554 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260925 14:51:11.372758 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:11.367968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58051 (local address 127.0.98.62:33581)
0925 14:51:11.368028 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:11.368030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:11.368154 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:51:11.368203 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:11.368203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:11.368214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:11.368269 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:11.368598 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.369188 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.369837 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.370057 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.370138 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.372650 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58051: BlockingRecv error: recv got EOF from 127.0.98.2:58051 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:11 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:11.492678 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:11.487905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:40885)
0925 14:51:11.488108 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:11.488122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:11.488217 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:11.488402 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:11.488403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:11.488599 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:11.488651 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.488653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.489353 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.489354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.489905 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:11.489908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.490287 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.490288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:11.490383 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.490844 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:11.490849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:11.492606 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260925 14:51:11.492799 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:11.487974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40885 (local address 127.0.98.62:33581)
0925 14:51:11.488093 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:11.488096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:11.488197 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:51:11.488289 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:11.488290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:11.488307 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:11.488380 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:11.488734 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.489261 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.489997 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:11.490197 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:11.490284 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:11.492703 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40885: BlockingRecv error: recv got EOF from 127.0.98.3:40885 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:12.255081 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:12.250580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:53587)
0925 14:51:12.250770 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:12.250783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:12.250845 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:12.250968 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:12.250969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:12.251162 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:12.251216 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.251224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.251845 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.251846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.252390 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.252392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.252737 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.252737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.252780 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.253208 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:12.253212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:12.255011 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
W20260925 14:51:12.255170 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:12.250670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53587 (local address 127.0.98.62:33581)
0925 14:51:12.250818 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:12.250820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:12.250842 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:51:12.250896 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:12.250896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:12.250906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:12.250963 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:12.251282 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.251766 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.252459 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.252672 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.252723 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.255097 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53587: BlockingRecv error: recv got EOF from 127.0.98.1:53587 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:12.377815 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:12.373055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:45271)
0925 14:51:12.373214 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:12.373225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:12.373285 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:12.373417 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:12.373418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:12.373575 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:12.373639 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.373640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.374400 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.374401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.374942 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.374945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.375297 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.375298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.375336 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.375820 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:12.375825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:12.377699 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260925 14:51:12.377856 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:12.373175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45271 (local address 127.0.98.62:33581)
0925 14:51:12.373236 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:12.373238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:12.373274 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:51:12.373329 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:12.373330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:12.373341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:12.373403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:12.373701 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.374271 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.375016 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.375233 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.375288 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.377791 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45271: BlockingRecv error: recv got EOF from 127.0.98.2:45271 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:12 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:12.497668 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:12.493101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:35091)
0925 14:51:12.493220 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:12.493236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:12.493291 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:12.493428 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:12.493428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:12.493584 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:12.493636 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.493637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.494318 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.494319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.494885 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:12.494887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.495360 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.495361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:12.495408 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.495916 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:12.495921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:12.497604 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
W20260925 14:51:12.497758 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:12.493171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35091 (local address 127.0.98.62:33581)
0925 14:51:12.493236 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:12.493238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:12.493281 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:51:12.493341 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:12.493341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:12.493354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:12.493406 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:12.493698 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.494246 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.494951 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:12.495266 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:12.495345 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:12.497690 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35091: BlockingRecv error: recv got EOF from 127.0.98.3:35091 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:13.260004 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:13.255551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:48435)
0925 14:51:13.255695 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:13.255713 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:13.255796 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:13.255952 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:13.255952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:13.256135 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:13.256182 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.256183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.256840 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.256841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.257383 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.257385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.257707 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.257708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.257745 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.258186 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:13.258191 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:13.259945 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
W20260925 14:51:13.260115 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:13.255713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48435 (local address 127.0.98.62:33581)
0925 14:51:13.255826 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:13.255827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:13.255835 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:13.255857 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:13.255857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:13.255866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:13.255926 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:13.256248 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.256777 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.257453 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.257647 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.257698 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.260045 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48435: BlockingRecv error: recv got EOF from 127.0.98.1:48435 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:13.383183 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:13.378307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:46095)
0925 14:51:13.378442 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:13.378454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:13.378518 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:13.378651 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:13.378652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:13.378882 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:13.378947 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.378949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.379768 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.379769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.380286 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.380288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.380607 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.380608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.380645 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.381107 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:13.381111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:13.383078 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
W20260925 14:51:13.383230 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:13.378364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46095 (local address 127.0.98.62:33581)
0925 14:51:13.378404 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:13.378406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:13.378511 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:51:13.378562 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:13.378562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:13.378573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:13.378637 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:13.379025 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.379697 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.380364 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.380532 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.380608 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.383163 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46095: BlockingRecv error: recv got EOF from 127.0.98.2:46095 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:13 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:13.502419 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:13.498110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:57299)
0925 14:51:13.498231 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:13.498242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:13.498334 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:13.498494 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:13.498495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:13.498656 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:13.498704 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.498705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.499302 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.499302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.499855 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:13.499856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.500150 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.500151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:13.500190 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.500605 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:13.500610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:13.502362 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
W20260925 14:51:13.502562 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:13.498171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57299 (local address 127.0.98.62:33581)
0925 14:51:13.498217 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:13.498219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:13.498321 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:51:13.498431 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:13.498432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:13.498442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:13.498489 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:13.498760 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.499240 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.499919 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:13.500092 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:13.500139 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:13.502483 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57299: BlockingRecv error: recv got EOF from 127.0.98.3:57299 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:14.265312 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:14.260479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:52453)
0925 14:51:14.260628 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:14.260640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:14.260702 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:14.260841 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:14.260842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:14.261065 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:14.261139 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.261141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.261811 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.261812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.262334 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.262336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.262709 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.262711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.262754 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.263197 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:14.263202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:14.265255 (+ 2053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
W20260925 14:51:14.265448 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:14.260587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52453 (local address 127.0.98.62:33581)
0925 14:51:14.260706 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:14.260708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:14.260720 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:14.260750 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:14.260751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:14.260763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:14.260821 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:14.261221 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.261722 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.262444 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.262645 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.262699 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.265357 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52453: BlockingRecv error: recv got EOF from 127.0.98.1:52453 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:14.391732 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:14.383644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:50829)
0925 14:51:14.383803 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:14.383814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:14.383884 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:14.384031 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:14.384032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:14.384193 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:14.384240 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.384241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.384911 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.384912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.385438 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.385440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.385780 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.385781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.385822 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.389733 (+ 3911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:14.389738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:14.391670 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
W20260925 14:51:14.391847 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:14.383719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50829 (local address 127.0.98.62:33581)
0925 14:51:14.383769 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:14.383772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:14.383888 (+ 116us) server_negotiation.cc:407] Connection header received
0925 14:51:14.383943 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:14.383943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:14.383957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:14.384004 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:14.384307 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.384807 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.385517 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.385713 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.385764 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.391776 (+ 6012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50829: BlockingRecv error: recv got EOF from 127.0.98.2:50829 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:14 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:14.507766 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:14.503006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:49937)
0925 14:51:14.503168 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:14.503205 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:14.503283 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:14.503449 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:14.503450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:14.503695 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:14.503753 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.503756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.504406 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.504407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.504950 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:14.504952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.505301 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.505302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:14.505342 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.505861 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:14.505867 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:14.507707 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
W20260925 14:51:14.507869 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:14.503099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49937 (local address 127.0.98.62:33581)
0925 14:51:14.503192 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:14.503194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:14.503283 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:51:14.503343 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:14.503344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:14.503363 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:14.503437 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:14.503822 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.504329 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.505046 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:14.505231 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:14.505292 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:14.507799 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49937: BlockingRecv error: recv got EOF from 127.0.98.3:49937 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:15.270526 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:15.265816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:36415)
0925 14:51:15.265959 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:15.265970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:15.266016 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:15.266191 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:15.266192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:15.266386 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:15.266435 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.266437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.267040 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.267040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.267707 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.267710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.268017 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.268018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.268053 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.268704 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:15.268709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:15.270469 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260925 14:51:15.270682 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:15.265928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36415 (local address 127.0.98.62:33581)
0925 14:51:15.266063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:15.266065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:15.266078 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:15.266099 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:15.266100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:15.266110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:15.266172 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:15.266491 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.266981 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.267775 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.267965 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.268018 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.270589 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36415: BlockingRecv error: recv got EOF from 127.0.98.1:36415 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:15.397068 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:15.392246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:39539)
0925 14:51:15.392414 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:15.392430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:15.392520 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:15.392687 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:15.392688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:15.392885 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:15.392934 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.392936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.393648 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.393650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.394230 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.394232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.394679 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.394680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.394738 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.395187 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:15.395192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:15.396944 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
W20260925 14:51:15.397082 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:15.392361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39539 (local address 127.0.98.62:33581)
0925 14:51:15.392453 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:15.392455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:15.392498 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:51:15.392613 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:15.392614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:15.392628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:15.392683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:15.393016 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.393542 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.394335 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.394589 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.394680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.397031 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39539: BlockingRecv error: recv got EOF from 127.0.98.2:39539 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:15 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:15.512811 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:15.508258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:48231)
0925 14:51:15.508400 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:15.508411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:15.508488 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:15.508742 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:15.508743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:15.508979 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:15.509053 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.509056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.509734 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.509735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.510267 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:15.510269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.510605 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.510606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:15.510647 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.511054 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:15.511058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:15.512754 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260925 14:51:15.512923 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:15.508369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48231 (local address 127.0.98.62:33581)
0925 14:51:15.508457 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:15.508459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:15.508472 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:15.508557 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:15.508559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:15.508573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:15.508636 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:15.509149 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.509664 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.510357 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:15.510540 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:15.510595 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:15.512841 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48231: BlockingRecv error: recv got EOF from 127.0.98.3:48231 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:16.275785 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:16.271038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:45759)
0925 14:51:16.271208 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:16.271220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:16.271287 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:16.271420 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:16.271421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:16.271609 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:16.271656 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.271658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.272275 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.272276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.272781 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.272782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.273203 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.273204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.273248 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.273763 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:16.273768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:16.275728 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
W20260925 14:51:16.275935 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:16.271131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45759 (local address 127.0.98.62:33581)
0925 14:51:16.271231 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:16.271233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:16.271274 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:51:16.271341 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:16.271342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:16.271351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:16.271399 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:16.271720 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.272212 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.272872 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.273122 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.273195 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.275825 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45759: BlockingRecv error: recv got EOF from 127.0.98.1:45759 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:16.403069 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:16.397699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:36555)
0925 14:51:16.397834 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:16.397848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:16.397953 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:16.398161 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:16.398162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:16.398390 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:16.398439 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.398441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.399183 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.399184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.399790 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.399792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.400211 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.400213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.400280 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.400862 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:16.400867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:16.402960 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
W20260925 14:51:16.403108 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:16.397731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36555 (local address 127.0.98.62:33581)
0925 14:51:16.397785 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:16.397788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:16.397932 (+ 144us) server_negotiation.cc:407] Connection header received
0925 14:51:16.398083 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:16.398084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:16.398098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:16.398155 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:16.398533 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.399093 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.399885 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.400117 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.400211 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.403044 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36555: BlockingRecv error: recv got EOF from 127.0.98.2:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:16 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:16.517854 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:16.513343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:48419)
0925 14:51:16.513485 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:16.513503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:16.513563 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:16.513698 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:16.513699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:16.513856 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:16.513906 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.513907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.514559 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.514559 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.515101 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:16.515102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.515417 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.515418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:16.515460 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.515894 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:16.515899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:16.517796 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
W20260925 14:51:16.517978 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:16.513408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48419 (local address 127.0.98.62:33581)
0925 14:51:16.513485 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:16.513487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:16.513568 (+ 81us) server_negotiation.cc:407] Connection header received
0925 14:51:16.513606 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:16.513607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:16.513619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:16.513676 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:16.513981 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.514495 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.515172 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:16.515356 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:16.515404 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:16.517880 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48419: BlockingRecv error: recv got EOF from 127.0.98.3:48419 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:17.280833 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:17.276321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:50625)
0925 14:51:17.276482 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:17.276494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:17.276572 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:17.276721 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:17.276722 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:17.276879 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:17.276925 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.276927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.277624 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.277625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.278150 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.278151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.278532 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.278533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.278573 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.279073 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:17.279099 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:17.280775 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
W20260925 14:51:17.280984 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:17.276431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50625 (local address 127.0.98.62:33581)
0925 14:51:17.276594 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:17.276596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:17.276606 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:17.276628 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:17.276628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:17.276639 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:17.276700 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:17.276995 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.277539 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.278232 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.278446 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.278523 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.280886 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50625: BlockingRecv error: recv got EOF from 127.0.98.1:50625 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":53,"threads_started":1}
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:17.408281 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:17.403731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:53743)
0925 14:51:17.403848 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:17.403865 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:17.403922 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:17.404065 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:17.404066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:17.404227 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:17.404274 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.404276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.404921 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.404922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.405470 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.405472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.405853 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.405854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.405911 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.406470 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:17.406475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:17.408200 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
W20260925 14:51:17.408373 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:17.403800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53743 (local address 127.0.98.62:33581)
0925 14:51:17.403870 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:17.403872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:17.403922 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:51:17.403971 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:17.403971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:17.403982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:17.404036 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:17.404339 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.404827 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.405559 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.405774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.405853 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.408289 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53743: BlockingRecv error: recv got EOF from 127.0.98.2:53743 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:17 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:17.523653 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:17.518301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:41415)
0925 14:51:17.518483 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:17.518496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:17.518589 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:17.518740 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:17.518741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:17.518921 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:17.518967 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.518968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.519660 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.519661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.520675 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:17.520677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.521172 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.521173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:17.521245 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.521697 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:17.521701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:17.523555 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
W20260925 14:51:17.523744 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:17.518428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41415 (local address 127.0.98.62:33581)
0925 14:51:17.518496 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:17.518498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:17.518570 (+ 72us) server_negotiation.cc:407] Connection header received
0925 14:51:17.518645 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:17.518646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:17.518661 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:17.518734 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:17.519051 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.519539 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.520779 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:17.521091 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:17.521161 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:17.523671 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41415: BlockingRecv error: recv got EOF from 127.0.98.3:41415 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:18.286188 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:18.281449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:39927)
0925 14:51:18.281595 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:18.281606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:18.281658 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:18.281801 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:18.281801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:18.281961 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:18.282012 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.282014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.282881 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.282882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.283449 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.283451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.283848 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.283849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.283913 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.284376 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:18.284381 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:18.286082 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260925 14:51:18.286242 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:18.281511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39927 (local address 127.0.98.62:33581)
0925 14:51:18.281647 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:18.281649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:18.281669 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:51:18.281697 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:18.281698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:18.281708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:18.281778 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:18.282078 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.282801 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.283551 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.283768 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.283846 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.286166 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39927: BlockingRecv error: recv got EOF from 127.0.98.1:39927 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:18.413625 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:18.408757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:54105)
0925 14:51:18.408922 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:18.408974 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:18.409052 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:18.409191 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:18.409192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:18.409389 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:18.409450 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.409452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.410100 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.410101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.410730 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.410733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.411059 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.411060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.411101 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.411551 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:18.411556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:18.413527 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260925 14:51:18.413678 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:18.408889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54105 (local address 127.0.98.62:33581)
0925 14:51:18.408945 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:18.408947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:18.409030 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:51:18.409108 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:18.409109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:18.409120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:18.409169 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:18.409511 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.410009 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.410808 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.410991 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.411043 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.413616 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54105: BlockingRecv error: recv got EOF from 127.0.98.2:54105 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:18 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:18.529008 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:18.524231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:44631)
0925 14:51:18.524398 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:18.524413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:18.524506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:18.524667 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:18.524668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:18.524849 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:18.524897 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.524899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.525585 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.525586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.526187 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:18.526189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.526619 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.526620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:18.526678 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.527185 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:18.527190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:18.528904 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
W20260925 14:51:18.529047 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:18.524327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44631 (local address 127.0.98.62:33581)
0925 14:51:18.524404 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:18.524406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:18.524484 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:51:18.524582 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:18.524583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:18.524598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:18.524657 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:18.524973 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.525490 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.526270 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:18.526530 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:18.526608 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:18.528983 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44631: BlockingRecv error: recv got EOF from 127.0.98.3:44631 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:19.290982 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:19.286732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:51299)
0925 14:51:19.286858 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:19.286868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:19.286917 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:19.287031 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:19.287031 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:19.287186 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:19.287230 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.287232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.287841 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.287841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.288374 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.288375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.288700 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.288701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.288738 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.289177 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:19.289182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:19.290923 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260925 14:51:19.291131 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:19.286807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51299 (local address 127.0.98.62:33581)
0925 14:51:19.286904 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:19.286906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:19.286918 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:19.286962 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:19.286962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:19.286972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:19.287025 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:19.287290 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.287777 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.288444 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.288630 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.288679 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.291044 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51299: BlockingRecv error: recv got EOF from 127.0.98.1:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":42,"threads_started":1}
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:19.422853 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:19.414131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:57841)
0925 14:51:19.414294 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:19.414305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:19.414396 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:19.414533 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:19.414534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:19.414701 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:19.414751 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.414753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.415418 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.415418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.416105 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.416108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.416607 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.416608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.416666 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.421036 (+ 4370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:19.421041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:19.422794 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
W20260925 14:51:19.423060 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:19.414230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57841 (local address 127.0.98.62:33581)
0925 14:51:19.414298 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:19.414300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:19.414367 (+ 67us) server_negotiation.cc:407] Connection header received
0925 14:51:19.414456 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:19.414457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:19.414470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:19.414512 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:19.414843 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.415348 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.416202 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.416524 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.416598 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.422998 (+ 6400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57841: BlockingRecv error: recv got EOF from 127.0.98.2:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:19 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:19.534053 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:19.529484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.3:54651)
0925 14:51:19.529608 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:19.529636 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:19.529708 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:19.529886 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:19.529887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:19.530069 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:19.530117 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.530119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.530880 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.530881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.531422 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:19.531423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.531806 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.531807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:19.531869 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.532300 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:19.532305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:19.533997 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
W20260925 14:51:19.534215 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:19.529585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54651 (local address 127.0.98.62:33581)
0925 14:51:19.529669 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:19.529671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:19.529685 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:51:19.529780 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:19.529781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:19.529796 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:19.529868 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:19.530195 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.530780 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.531509 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:19.531719 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:19.531805 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:19.534147 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54651: BlockingRecv error: recv got EOF from 127.0.98.3:54651 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:51:20 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:20 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:20.296397 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.291504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.1:36589)
0925 14:51:20.291663 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:20.291677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:20.291748 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:20.291911 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:20.291912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:20.292093 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:20.292141 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.292143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.292839 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.292841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.293438 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.293440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.293822 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.293823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.293870 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.294449 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:20.294454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:20.296325 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
W20260925 14:51:20.296528 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:20.291622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36589 (local address 127.0.98.62:33581)
0925 14:51:20.291758 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:20.291759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:20.291767 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:20.291803 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:20.291804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:20.291814 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:20.291866 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:20.292221 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.292745 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.293538 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.293749 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.293805 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.296420 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36589: BlockingRecv error: recv got EOF from 127.0.98.1:36589 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
Sep 25 14:51:20 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:51:20 dist-test-slave-gpc1 krb5kdc[8537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:51:20.428292 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.423395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.98.2:58147)
0925 14:51:20.423558 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:20.423569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:20.423639 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:20.423760 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:20.423761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:20.423917 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:20.423964 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.423965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.424561 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.424561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.425147 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.425150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.425460 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.425460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.425502 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.425927 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:20.425931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:20.428185 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:33581: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
W20260925 14:51:20.428321 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:20.423508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58147 (local address 127.0.98.62:33581)
0925 14:51:20.423564 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:20.423566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:20.423637 (+ 71us) server_negotiation.cc:407] Connection header received
0925 14:51:20.423684 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:20.423684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:20.423694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:20.423743 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:20.424020 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.424499 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.425215 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.425398 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.425452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.428258 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58147: BlockingRecv error: recv got EOF from 127.0.98.2:58147 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:51:20.449791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9120
I20260925 14:51:20.454380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9272
I20260925 14:51:20.458737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9404
I20260925 14:51:20.463001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9050
2026-09-25T14:51:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:20Z Disabled control of system clock
I20260925 14:51:20.490664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46311
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:51:20.570101 9601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:20.570384 9601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:20.570421 9601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:20.571672 9601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:20.571743 9601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:20.571765 9601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:20.571785 9601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:20.571816 9601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:20.573165 9601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:20.573438 9601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:20.573733 9601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:20.578491 9606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.578474 9609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.578661 9607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.579105 9601 server_base.cc:1034] running on GCE node
I20260925 14:51:20.579360 9601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:20.579644 9601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:20.580811 9601 hybrid_clock.cc:648] HybridClock initialized: now 1790347880580812 us; error 40 us; skew 500 ppm
I20260925 14:51:20.582103 9601 webserver.cc:460] Webserver started at http://127.0.98.62:37831/ using document root <none> and password file <none>
I20260925 14:51:20.582420 9601 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:20.582505 9601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:20.582674 9601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:20.583818 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e54471cac5304f32abbaed42a7ab9444"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "b4dcfc693b3d42f2ed26c83b7f349ab6"
server_key_iv: "aee7364b422cb43c254db4c91c2523d4"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.584173 9601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e54471cac5304f32abbaed42a7ab9444"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "b4dcfc693b3d42f2ed26c83b7f349ab6"
server_key_iv: "aee7364b422cb43c254db4c91c2523d4"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.585480 9601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:20.586195 9615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.586402 9601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:20.586505 9601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e54471cac5304f32abbaed42a7ab9444"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "b4dcfc693b3d42f2ed26c83b7f349ab6"
server_key_iv: "aee7364b422cb43c254db4c91c2523d4"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.586599 9601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:20.600736 9601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:20.601054 9601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:20.601210 9601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:20.605378 9601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46311
I20260925 14:51:20.605417 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46311 every 8 connection(s)
I20260925 14:51:20.605764 9601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:20.606465 9668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:20.609171 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444: Bootstrap starting.
I20260925 14:51:20.609793 9668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:20.610110 9668 log.cc:824] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:20.610893 9668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444: No bootstrap required, opened a new log
I20260925 14:51:20.612496 9668 raft_consensus.cc:348] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } }
I20260925 14:51:20.612640 9668 raft_consensus.cc:374] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:20.612663 9668 raft_consensus.cc:729] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e54471cac5304f32abbaed42a7ab9444, State: Initialized, Role: FOLLOWER
I20260925 14:51:20.612910 9668 consensus_queue.cc:260] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } }
I20260925 14:51:20.612998 9668 raft_consensus.cc:388] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:20.613041 9668 raft_consensus.cc:482] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:20.613095 9668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:20.613639 9668 raft_consensus.cc:504] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } }
I20260925 14:51:20.613752 9668 leader_election.cc:302] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e54471cac5304f32abbaed42a7ab9444; no voters:
I20260925 14:51:20.613922 9668 leader_election.cc:288] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:20.613967 9671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:20.614168 9671 raft_consensus.cc:686] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [term 1 LEADER]: Becoming Leader. State: Replica: e54471cac5304f32abbaed42a7ab9444, State: Running, Role: LEADER
I20260925 14:51:20.614213 9668 sys_catalog.cc:564] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:20.614305 9671 consensus_queue.cc:237] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } }
I20260925 14:51:20.614713 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e54471cac5304f32abbaed42a7ab9444. Latest consensus state: current_term: 1 leader_uuid: "e54471cac5304f32abbaed42a7ab9444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } } }
I20260925 14:51:20.614848 9672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e54471cac5304f32abbaed42a7ab9444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e54471cac5304f32abbaed42a7ab9444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46311 } } }
I20260925 14:51:20.614866 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:20.615074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9601
I20260925 14:51:20.615152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:20.615173 9672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:20.615185 9680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:20.615383 392 external_mini_cluster.cc:1428] Setting key 9ef6d643111768d8c70ce211551eb09c
I20260925 14:51:20.615713 9680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:20.617789 9680 catalog_manager.cc:1347] Generated new cluster ID: 757413c6d5844a28ad4c4887ca39f717
I20260925 14:51:20.617842 9680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:20.622189 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.616005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.98.62:46311)
0925 14:51:20.616432 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:20.616442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:20.616479 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:51:20.616600 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:20.616605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:20.616658 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:20.616800 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:20.617028 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.618398 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.619555 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.620217 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.620337 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.621311 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:20.621315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:20.621316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:20.621520 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:20.621561 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:20.621578 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:20.621642 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:20.621886 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:51:20.621965 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":59,"threads_started":1}
I20260925 14:51:20.623493 9680 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:20.624145 9680 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:20.628970 9680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e54471cac5304f32abbaed42a7ab9444: Generated new TSK 0
I20260925 14:51:20.629160 9680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:51:20.635454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--builtin_ntp_servers=127.0.98.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:51:20.718839 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:20.719051 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:20.719074 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:20.720243 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:20.720285 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:20.720311 9692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:20.721697 9692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:20.721951 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:20.722246 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:20.728127 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.728332 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.728374 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.728605 9692 server_base.cc:1034] running on GCE node
I20260925 14:51:20.728826 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:20.729072 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:20.730273 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790347880730256 us; error 32 us; skew 500 ppm
I20260925 14:51:20.731747 9692 webserver.cc:460] Webserver started at http://127.0.98.1:42753/ using document root <none> and password file <none>
I20260925 14:51:20.732000 9692 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:20.732079 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:20.732225 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:20.733307 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "7168d24bf6f74aac811cc62be62feaeb"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "f3de8872658db6e1983804faed9a4202"
server_key_iv: "36b65d144630f20a21501fbb30e3208a"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.733690 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7168d24bf6f74aac811cc62be62feaeb"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "f3de8872658db6e1983804faed9a4202"
server_key_iv: "36b65d144630f20a21501fbb30e3208a"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.735204 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:20.736001 9706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.736203 9692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:20.736308 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "7168d24bf6f74aac811cc62be62feaeb"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "f3de8872658db6e1983804faed9a4202"
server_key_iv: "36b65d144630f20a21501fbb30e3208a"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.736382 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:20.746692 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:20.746991 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:20.747139 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:20.747361 9692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:20.747684 9692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:20.747717 9692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.747738 9692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:20.747783 9692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.753496 9692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44997
I20260925 14:51:20.753608 9819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44997 every 8 connection(s)
I20260925 14:51:20.753916 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:20.759282 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.754699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46969 (local address 127.0.98.62:46311)
0925 14:51:20.754747 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:20.754750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:20.754963 (+ 213us) server_negotiation.cc:407] Connection header received
0925 14:51:20.755519 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:20.755521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:20.755545 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:20.755599 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:20.756302 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.756813 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.757815 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.758027 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.758107 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.758948 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:20.758949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:20.758950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:20.758955 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:20.758970 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:20.758974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:20.759030 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:20.759170 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 14:51:20.759202 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:20.759526 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.754670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46311 (local address 127.0.98.1:46969)
0925 14:51:20.754843 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:20.754875 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:20.755407 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:20.755683 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:20.755692 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:20.755700 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:20.756201 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.756205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.756912 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.756914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.757720 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.757722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.758114 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.758116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.758250 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.758810 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:20.758811 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:20.758838 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:20.758861 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:20.759086 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:20.759090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:20.759093 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:20.759174 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:51:20.759279 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260925 14:51:20.760035 9820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46311
I20260925 14:51:20.760131 9820 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:20.760316 9820 heartbeater.cc:511] Master 127.0.98.62:46311 requested a full tablet report, sending...
I20260925 14:51:20.760815 9632 ts_manager.cc:194] Registered new tserver with Master: 7168d24bf6f74aac811cc62be62feaeb (127.0.98.1:44997)
I20260925 14:51:20.761381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9692
I20260925 14:51:20.761385 9632 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46969
I20260925 14:51:20.761454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:20.761665 392 external_mini_cluster.cc:1428] Setting key d9f4a2584fa79ccbb2122ed0c7b06828
I20260925 14:51:20.762624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--builtin_ntp_servers=127.0.98.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:51:20.844616 9823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:20.844823 9823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:20.844846 9823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:20.846012 9823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:20.846055 9823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:20.846081 9823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:20.847510 9823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:20.847764 9823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:20.848066 9823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:20.853293 9831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.853425 9823 server_base.cc:1034] running on GCE node
W20260925 14:51:20.853307 9828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.853381 9829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.853699 9823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:20.853924 9823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:20.855070 9823 hybrid_clock.cc:648] HybridClock initialized: now 1790347880855052 us; error 39 us; skew 500 ppm
I20260925 14:51:20.856376 9823 webserver.cc:460] Webserver started at http://127.0.98.2:45883/ using document root <none> and password file <none>
I20260925 14:51:20.856698 9823 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:20.856747 9823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:20.856897 9823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:20.858052 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "5bfc695cafdc4aaa94dc8b2635805429"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "352118c7077dccdbdc010c7f901c8c68"
server_key_iv: "9a769f29d45f2e5e733740c806ce9906"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.858489 9823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5bfc695cafdc4aaa94dc8b2635805429"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "352118c7077dccdbdc010c7f901c8c68"
server_key_iv: "9a769f29d45f2e5e733740c806ce9906"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.859839 9823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:20.860608 9837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.860781 9823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:20.860879 9823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "5bfc695cafdc4aaa94dc8b2635805429"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "352118c7077dccdbdc010c7f901c8c68"
server_key_iv: "9a769f29d45f2e5e733740c806ce9906"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.860961 9823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:20.872392 9823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:20.872766 9823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:20.872933 9823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:20.873219 9823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:20.873621 9823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:20.873674 9823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.873713 9823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:20.873739 9823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.881435 9823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44915
I20260925 14:51:20.881536 9950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44915 every 8 connection(s)
I20260925 14:51:20.881837 9823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:20.887204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9823
I20260925 14:51:20.887269 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.882670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58245 (local address 127.0.98.62:46311)
0925 14:51:20.882736 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:20.882738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:20.883012 (+ 274us) server_negotiation.cc:407] Connection header received
0925 14:51:20.883513 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:20.883514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:20.883525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:20.883592 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:20.884263 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.884769 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.885735 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.885925 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.885984 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.886898 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:20.886899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:20.886900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:20.886904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:20.886919 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:20.886923 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:20.887033 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:20.887138 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:51:20.887170 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:51:20.887295 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:20.887429 9953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:20.882662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46311 (local address 127.0.98.2:58245)
0925 14:51:20.882886 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:20.882921 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:20.883404 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:20.883648 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:20.883652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:20.883660 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:20.884182 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.884187 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.884846 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.884847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.885653 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:20.885655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:20.886000 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:20.886001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:20.886124 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:20.886753 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:20.886753 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:20.886782 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:20.886805 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:20.887038 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:20.887061 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:20.887063 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:20.887116 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:51:20.887193 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":107,"threads_started":1}
I20260925 14:51:20.887517 392 external_mini_cluster.cc:1428] Setting key 1f0b32ed2d57e6f1f62b2655ba36a642
I20260925 14:51:20.887946 9951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46311
I20260925 14:51:20.888043 9951 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:20.888232 9951 heartbeater.cc:511] Master 127.0.98.62:46311 requested a full tablet report, sending...
I20260925 14:51:20.888622 9631 ts_manager.cc:194] Registered new tserver with Master: 5bfc695cafdc4aaa94dc8b2635805429 (127.0.98.2:44915)
I20260925 14:51:20.888698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--builtin_ntp_servers=127.0.98.20:46669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:51:20.889104 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58245
W20260925 14:51:20.971676 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:20.971921 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:20.971956 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:20.973096 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:20.973157 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:20.973196 9954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:20.974562 9954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:20.974828 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:20.975148 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:20.980273 9959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:20.980242 9960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.980427 9954 server_base.cc:1034] running on GCE node
W20260925 14:51:20.980244 9962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:20.980696 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:20.980971 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:20.982137 9954 hybrid_clock.cc:648] HybridClock initialized: now 1790347880982120 us; error 29 us; skew 500 ppm
I20260925 14:51:20.983523 9954 webserver.cc:460] Webserver started at http://127.0.98.3:34663/ using document root <none> and password file <none>
I20260925 14:51:20.983777 9954 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:20.983846 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:20.983980 9954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:20.985037 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "c67ce27d5fd5431c9e282c25270f62af"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "879ec4049af4f724048c8738a96b0e85"
server_key_iv: "16f72d9489e4f3a7936d5a509e0f9f9b"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.985396 9954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c67ce27d5fd5431c9e282c25270f62af"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "879ec4049af4f724048c8738a96b0e85"
server_key_iv: "16f72d9489e4f3a7936d5a509e0f9f9b"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.986775 9954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:20.987516 9968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:20.987710 9954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:20.987808 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "c67ce27d5fd5431c9e282c25270f62af"
format_stamp: "Formatted at 2026-09-25 14:51:20 on dist-test-slave-gpc1"
server_key: "879ec4049af4f724048c8738a96b0e85"
server_key_iv: "16f72d9489e4f3a7936d5a509e0f9f9b"
server_key_version: "encryptionkey@0"
I20260925 14:51:20.987891 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:21.005820 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:21.006162 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:21.006319 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:21.006588 9954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:21.006927 9954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:21.006985 9954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.007026 9954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:21.007056 9954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.012766 9954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38685
I20260925 14:51:21.012828 10081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38685 every 8 connection(s)
I20260925 14:51:21.013217 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:21.013780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 9954
I20260925 14:51:21.013878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:21.014139 392 external_mini_cluster.cc:1428] Setting key adb4ee2eb0dedd0e2ea6ad12834124af
I20260925 14:51:21.019449 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.014153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43719 (local address 127.0.98.62:46311)
0925 14:51:21.014248 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.014250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.014593 (+ 343us) server_negotiation.cc:407] Connection header received
0925 14:51:21.015567 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.015568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.015581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.015622 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.016397 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:21.016918 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:21.017933 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:21.018155 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:21.018232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:21.019151 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:21.019153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:21.019153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:21.019158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:21.019175 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:21.019179 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:21.019250 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:21.019330 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:51:21.019362 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:51:21.019665 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.014066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46311 (local address 127.0.98.3:43719)
0925 14:51:21.014421 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:21.014472 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:21.015452 (+ 980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:21.015709 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:21.015713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:21.015729 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:21.016290 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:21.016295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:21.017009 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:21.017010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:21.017826 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:21.017828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:21.018249 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:21.018250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:21.018464 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:21.019010 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:21.019010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:21.019039 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:21.019062 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:21.019252 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:21.019257 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:21.019261 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:21.019332 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:51:21.019435 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260925 14:51:21.020155 10082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46311
I20260925 14:51:21.020233 10082 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:21.020370 10082 heartbeater.cc:511] Master 127.0.98.62:46311 requested a full tablet report, sending...
I20260925 14:51:21.020679 9631 ts_manager.cc:194] Registered new tserver with Master: c67ce27d5fd5431c9e282c25270f62af (127.0.98.3:38685)
I20260925 14:51:21.021045 9631 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43719
I20260925 14:51:21.026127 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:51:21.028075 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:21.027598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.98.62:46311)
0925 14:51:21.027667 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.027669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.027756 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:51:21.027808 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.027809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.027819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.027870 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.028007 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50478: BlockingRecv error: recv got EOF from 127.0.0.1:50478 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:51:21.028156 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9692
I20260925 14:51:21.032892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9823
I20260925 14:51:21.037333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9954
I20260925 14:51:21.041483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 9601
2026-09-25T14:51:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (577 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:21Z Disabled control of system clock
I20260925 14:51:21.067857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:51:21.146787 10093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:21.146981 10093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:21.147001 10093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:21.148080 10093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:21.148114 10093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:21.148128 10093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:21.148140 10093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:21.148152 10093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:21.149400 10093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:21.149587 10093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:21.149802 10093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:21.154575 10098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:21.154610 10101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.154657 10093 server_base.cc:1034] running on GCE node
W20260925 14:51:21.154611 10099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.154994 10093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:21.155305 10093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:21.156476 10093 hybrid_clock.cc:648] HybridClock initialized: now 1790347881156455 us; error 39 us; skew 500 ppm
I20260925 14:51:21.157823 10093 webserver.cc:460] Webserver started at http://127.0.98.62:37537/ using document root <none> and password file <none>
I20260925 14:51:21.158069 10093 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:21.158141 10093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:21.158293 10093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:21.159394 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e2a96e12c6bc4baa9b72eb7429c110cc"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "ba835485dcae1bc71ffae2b974f7e38c"
server_key_iv: "23bf02ea24cfc3e56fc8c2fc86a46a91"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.159765 10093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e2a96e12c6bc4baa9b72eb7429c110cc"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "ba835485dcae1bc71ffae2b974f7e38c"
server_key_iv: "23bf02ea24cfc3e56fc8c2fc86a46a91"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.161091 10093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:21.161903 10107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.162083 10093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:21.162215 10093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e2a96e12c6bc4baa9b72eb7429c110cc"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "ba835485dcae1bc71ffae2b974f7e38c"
server_key_iv: "23bf02ea24cfc3e56fc8c2fc86a46a91"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.162315 10093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:21.166160 10093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:21.166522 10093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:21.166697 10093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:21.171218 10093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42961
I20260925 14:51:21.171248 10159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42961 every 8 connection(s)
I20260925 14:51:21.171634 10093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:21.172277 10160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:21.175015 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc: Bootstrap starting.
I20260925 14:51:21.175535 10160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:21.175777 10160 log.cc:824] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:21.176399 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc: No bootstrap required, opened a new log
I20260925 14:51:21.177867 10160 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260925 14:51:21.178017 10160 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:21.178040 10160 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a96e12c6bc4baa9b72eb7429c110cc, State: Initialized, Role: FOLLOWER
I20260925 14:51:21.178138 10160 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260925 14:51:21.178215 10160 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:21.178251 10160 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:21.178319 10160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:21.178867 10160 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260925 14:51:21.179008 10160 leader_election.cc:302] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a96e12c6bc4baa9b72eb7429c110cc; no voters:
I20260925 14:51:21.179180 10160 leader_election.cc:288] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:21.179270 10163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:21.179417 10163 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [term 1 LEADER]: Becoming Leader. State: Replica: e2a96e12c6bc4baa9b72eb7429c110cc, State: Running, Role: LEADER
I20260925 14:51:21.179461 10160 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:21.179514 10163 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } }
I20260925 14:51:21.179914 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a96e12c6bc4baa9b72eb7429c110cc. Latest consensus state: current_term: 1 leader_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260925 14:51:21.179996 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:21.180164 10164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a96e12c6bc4baa9b72eb7429c110cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } }
I20260925 14:51:21.180222 10172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:21.180227 10164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:21.180769 10172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:21.181790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10093
I20260925 14:51:21.181905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:21.182094 392 external_mini_cluster.cc:1428] Setting key 90a97eaff68431ed35d0c8935eddc9a6
I20260925 14:51:21.182893 10172 catalog_manager.cc:1347] Generated new cluster ID: 06f279047aa14da4a80f0878854322d8
I20260925 14:51:21.182945 10172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:21.184284 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.182585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.0.98.62:42961)
0925 14:51:21.182765 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.182774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.182852 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:51:21.182992 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.182998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.183053 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.183196 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.183833 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:21.183837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:21.183838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:21.183842 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:21.183888 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:21.183902 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:21.183973 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:21.184051 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:51:21.184109 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260925 14:51:21.192935 10172 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:21.193099 10172 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:21.198184 10172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a96e12c6bc4baa9b72eb7429c110cc: Generated new TSK 0
I20260925 14:51:21.198412 10172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:51:21.201504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--builtin_ntp_servers=127.0.98.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:51:21.285889 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:21.286092 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:21.286115 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:21.287338 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:21.287379 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:21.287405 10184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:21.288754 10184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:21.289011 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:21.289304 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:21.294283 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.294376 10184 server_base.cc:1034] running on GCE node
W20260925 14:51:21.294276 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:21.294502 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.294736 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:21.295014 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:21.296192 10184 hybrid_clock.cc:648] HybridClock initialized: now 1790347881296168 us; error 41 us; skew 500 ppm
I20260925 14:51:21.297626 10184 webserver.cc:460] Webserver started at http://127.0.98.1:33249/ using document root <none> and password file <none>
I20260925 14:51:21.297885 10184 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:21.297942 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:21.298089 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:21.299513 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "ebcc0562935542f28eaaa443cc93746a"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "c54944281c24f3ef3327a81d81c72392"
server_key_iv: "f80db287f1ba71145e93386defe6e646"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.299959 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ebcc0562935542f28eaaa443cc93746a"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "c54944281c24f3ef3327a81d81c72392"
server_key_iv: "f80db287f1ba71145e93386defe6e646"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.301699 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:21.302757 10198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.303032 10184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:21.303135 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "ebcc0562935542f28eaaa443cc93746a"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "c54944281c24f3ef3327a81d81c72392"
server_key_iv: "f80db287f1ba71145e93386defe6e646"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.303201 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:21.307652 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:21.307916 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:21.308032 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:21.308228 10184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:21.308516 10184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:21.308554 10184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.308580 10184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:21.308597 10184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.315443 10184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40639
I20260925 14:51:21.315490 10311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40639 every 8 connection(s)
I20260925 14:51:21.315824 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:21.316280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10184
I20260925 14:51:21.316373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:21.316630 392 external_mini_cluster.cc:1428] Setting key ef636e02360ed9c5190d8237abed09b8
I20260925 14:51:21.317705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--builtin_ntp_servers=127.0.98.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 14:51:21.319439 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.316647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53155 (local address 127.0.98.62:42961)
0925 14:51:21.316697 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.316699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.317034 (+ 335us) server_negotiation.cc:407] Connection header received
0925 14:51:21.317657 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.317658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.317663 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.317701 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.319151 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:21.319153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:21.319153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:21.319158 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:21.319177 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:21.319183 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:21.319218 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:21.319346 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:51:21.319362 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:51:21.319631 10314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.316646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.98.1:53155)
0925 14:51:21.316844 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:21.316890 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:21.317560 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:21.317788 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:21.317794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:21.317805 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:21.318950 (+ 1145us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:21.318950 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:21.318990 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:21.319030 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:21.319262 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:21.319267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:21.319270 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:21.319345 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:51:21.319408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260925 14:51:21.320061 10312 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42961
I20260925 14:51:21.320155 10312 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:21.320307 10312 heartbeater.cc:511] Master 127.0.98.62:42961 requested a full tablet report, sending...
I20260925 14:51:21.320794 10124 ts_manager.cc:194] Registered new tserver with Master: ebcc0562935542f28eaaa443cc93746a (127.0.98.1:40639)
W20260925 14:51:21.398739 10315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:21.398942 10315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:21.398965 10315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:21.400142 10315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:21.400182 10315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:21.400208 10315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:21.401559 10315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:21.401813 10315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:21.402105 10315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:21.407189 10321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:21.407193 10320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.407260 10315 server_base.cc:1034] running on GCE node
W20260925 14:51:21.407218 10323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.407577 10315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:21.407825 10315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:21.408977 10315 hybrid_clock.cc:648] HybridClock initialized: now 1790347881408936 us; error 39 us; skew 500 ppm
I20260925 14:51:21.410128 10315 webserver.cc:460] Webserver started at http://127.0.98.2:38265/ using document root <none> and password file <none>
I20260925 14:51:21.410321 10315 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:21.410424 10315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:21.410539 10315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:21.411569 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "7fef2101280347f5beb098b30a8d8714"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "4cad2f39dc86c24252c2f9409460e8e2"
server_key_iv: "5c790929802c6ebb55b6b7ebf4de78e4"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.411865 10315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7fef2101280347f5beb098b30a8d8714"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "4cad2f39dc86c24252c2f9409460e8e2"
server_key_iv: "5c790929802c6ebb55b6b7ebf4de78e4"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.413089 10315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:21.413785 10329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.413965 10315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:21.414033 10315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "7fef2101280347f5beb098b30a8d8714"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "4cad2f39dc86c24252c2f9409460e8e2"
server_key_iv: "5c790929802c6ebb55b6b7ebf4de78e4"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.414083 10315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:21.417459 10315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:21.417670 10315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:21.417763 10315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:21.417904 10315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:21.418133 10315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:21.418162 10315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.418182 10315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:21.418195 10315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.423913 10315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35167
I20260925 14:51:21.423949 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35167 every 8 connection(s)
I20260925 14:51:21.424269 10315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:21.427162 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.425008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46585 (local address 127.0.98.62:42961)
0925 14:51:21.425089 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.425091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.425293 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:51:21.425732 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.425733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.425738 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.425805 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.426945 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:21.426946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:21.426946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:21.426950 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:21.426967 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:21.426970 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:21.427020 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:21.427097 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:51:21.427110 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:51:21.427323 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.425005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.98.2:46585)
0925 14:51:21.425167 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:21.425206 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:21.425652 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:21.425855 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:21.425858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:21.425866 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:21.426799 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:21.426800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:21.426836 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:21.426864 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:21.427018 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:21.427021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:21.427023 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:21.427074 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:51:21.427121 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260925 14:51:21.427788 10443 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42961
I20260925 14:51:21.427865 10443 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:21.428021 10443 heartbeater.cc:511] Master 127.0.98.62:42961 requested a full tablet report, sending...
I20260925 14:51:21.428375 10124 ts_manager.cc:194] Registered new tserver with Master: 7fef2101280347f5beb098b30a8d8714 (127.0.98.2:35167)
I20260925 14:51:21.433156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10315
I20260925 14:51:21.433235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:21.433380 392 external_mini_cluster.cc:1428] Setting key 66870513f6ace86878e8d36abe4ac2c8
I20260925 14:51:21.434322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--builtin_ntp_servers=127.0.98.20:46255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:51:21.515700 10446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:21.515902 10446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:21.515924 10446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:21.517094 10446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:21.517133 10446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:21.517159 10446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:21.518611 10446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:21.518792 10446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:21.519026 10446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:21.524137 10454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:21.524276 10452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.524331 10446 server_base.cc:1034] running on GCE node
W20260925 14:51:21.524144 10451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:21.524641 10446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:21.524928 10446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:21.526081 10446 hybrid_clock.cc:648] HybridClock initialized: now 1790347881526060 us; error 35 us; skew 500 ppm
I20260925 14:51:21.527427 10446 webserver.cc:460] Webserver started at http://127.0.98.3:33089/ using document root <none> and password file <none>
I20260925 14:51:21.527690 10446 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:21.527738 10446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:21.527889 10446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:21.528975 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "3449d1b0e35f469cbe4f7bd9375771fb"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "9da9e319297428c2576795aad886fc69"
server_key_iv: "4294ef25701a634460b20c5a332d5d02"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.529330 10446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3449d1b0e35f469cbe4f7bd9375771fb"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "9da9e319297428c2576795aad886fc69"
server_key_iv: "4294ef25701a634460b20c5a332d5d02"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.530682 10446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:21.531383 10460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.531563 10446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.531656 10446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "3449d1b0e35f469cbe4f7bd9375771fb"
format_stamp: "Formatted at 2026-09-25 14:51:21 on dist-test-slave-gpc1"
server_key: "9da9e319297428c2576795aad886fc69"
server_key_iv: "4294ef25701a634460b20c5a332d5d02"
server_key_version: "encryptionkey@0"
I20260925 14:51:21.531726 10446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:21.535006 10446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:21.535266 10446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:21.535410 10446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:21.535625 10446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:21.535912 10446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:21.535964 10446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.536001 10446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:21.536027 10446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:21.541706 10446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37921
I20260925 14:51:21.541777 10573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37921 every 8 connection(s)
I20260925 14:51:21.542102 10446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:21.545199 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.542851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42367 (local address 127.0.98.62:42961)
0925 14:51:21.542913 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.542915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.543158 (+ 243us) server_negotiation.cc:407] Connection header received
0925 14:51:21.543656 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.543657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.543663 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.543709 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.544930 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:21.544932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:21.544932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:51:21.544936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:21.544952 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:51:21.544956 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:21.545010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:21.545112 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:51:21.545129 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:51:21.545392 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:21.542832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.98.3:42367)
0925 14:51:21.543038 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:21.543072 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:21.543552 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:21.543774 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:21.543777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:21.543787 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:21.544783 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:51:21.544783 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:21.544811 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:51:21.544838 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:21.545019 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:21.545024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:21.545027 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:21.545120 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:51:21.545181 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260925 14:51:21.545881 10574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42961
I20260925 14:51:21.545953 10574 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:21.546119 10574 heartbeater.cc:511] Master 127.0.98.62:42961 requested a full tablet report, sending...
I20260925 14:51:21.546489 10124 ts_manager.cc:194] Registered new tserver with Master: 3449d1b0e35f469cbe4f7bd9375771fb (127.0.98.3:37921)
I20260925 14:51:21.548907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10446
I20260925 14:51:21.548978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:21.549126 392 external_mini_cluster.cc:1428] Setting key b783c933035e02e87d4dbf80f2acd643
I20260925 14:51:21.550285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:51:21.552223 10183 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:21.551673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.0.98.62:42961)
0925 14:51:21.551748 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:21.551751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:21.551870 (+ 119us) server_negotiation.cc:407] Connection header received
0925 14:51:21.551949 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:21.551950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:21.551955 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:21.552002 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:21.552143 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40348: BlockingRecv error: recv got EOF from 127.0.0.1:40348 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:51:21.552351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10184
I20260925 14:51:21.557197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10315
I20260925 14:51:21.561406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10446
I20260925 14:51:21.565527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10093
2026-09-25T14:51:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:21 dist-test-slave-gpc1 krb5kdc[10584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:21 dist-test-slave-gpc1 krb5kdc[10584](info): set up 2 sockets
Sep 25 14:51:21 dist-test-slave-gpc1 krb5kdc[10584](info): commencing operation
krb5kdc: starting...
W20260925 14:51:22.623260 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:22 dist-test-slave-gpc1 krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:22.747687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:22.829870 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:22.830144 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:22.830168 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:22.831346 10597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:22.831394 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:22.831415 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:22.831430 10597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:22.831444 10597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:22.832943 10597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:22.833137 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:22.833323 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:22.838163 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:22.838217 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:22.838188 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:22.838371 10597 server_base.cc:1034] running on GCE node
I20260925 14:51:22.838624 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:22.838905 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:22.840083 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790347882840070 us; error 28 us; skew 500 ppm
Sep 25 14:51:22 dist-test-slave-gpc1 krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:22.842522 10597 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:22.842897 10597 webserver.cc:460] Webserver started at http://127.0.98.62:33447/ using document root <none> and password file <none>
I20260925 14:51:22.843101 10597 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:22.843143 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:22.843231 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:22.844199 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "cc5efdd9de86411e8646d87eb7e74b79"
format_stamp: "Formatted at 2026-09-25 14:51:22 on dist-test-slave-gpc1"
server_key: "9bae45d7580d2684209ada298dc7b3d5"
server_key_iv: "21a3cadeaa30d8db5e65b980299e47bf"
server_key_version: "encryptionkey@0"
I20260925 14:51:22.844488 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc5efdd9de86411e8646d87eb7e74b79"
format_stamp: "Formatted at 2026-09-25 14:51:22 on dist-test-slave-gpc1"
server_key: "9bae45d7580d2684209ada298dc7b3d5"
server_key_iv: "21a3cadeaa30d8db5e65b980299e47bf"
server_key_version: "encryptionkey@0"
I20260925 14:51:22.845741 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:22.846441 10612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:22.846612 10597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:22.846698 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "cc5efdd9de86411e8646d87eb7e74b79"
format_stamp: "Formatted at 2026-09-25 14:51:22 on dist-test-slave-gpc1"
server_key: "9bae45d7580d2684209ada298dc7b3d5"
server_key_iv: "21a3cadeaa30d8db5e65b980299e47bf"
server_key_version: "encryptionkey@0"
I20260925 14:51:22.846769 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:22.888226 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:22.890782 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:22.890945 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:22.894968 10597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44027
I20260925 14:51:22.895022 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44027 every 8 connection(s)
I20260925 14:51:22.895375 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:22.896406 10665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:22.898674 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: Bootstrap starting.
I20260925 14:51:22.899194 10665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:22.899418 10665 log.cc:824] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:22.900069 10665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: No bootstrap required, opened a new log
I20260925 14:51:22.901535 10665 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } }
I20260925 14:51:22.901664 10665 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:22.901685 10665 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5efdd9de86411e8646d87eb7e74b79, State: Initialized, Role: FOLLOWER
I20260925 14:51:22.901793 10665 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } }
I20260925 14:51:22.901872 10665 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:22.901908 10665 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:22.901968 10665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:22.902781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10597
I20260925 14:51:22.902851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:22.902850 10665 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } }
I20260925 14:51:22.902962 10665 leader_election.cc:302] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc5efdd9de86411e8646d87eb7e74b79; no voters:
I20260925 14:51:22.903041 392 external_mini_cluster.cc:1428] Setting key b1846ffd72270cae0ab0f003a7ed99ff
I20260925 14:51:22.903144 10665 leader_election.cc:288] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:22.903170 10668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:22.903271 10668 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [term 1 LEADER]: Becoming Leader. State: Replica: cc5efdd9de86411e8646d87eb7e74b79, State: Running, Role: LEADER
I20260925 14:51:22.903349 10668 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } }
I20260925 14:51:22.903399 10665 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:22.903827 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc5efdd9de86411e8646d87eb7e74b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } } }
I20260925 14:51:22.903892 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:22.904039 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc5efdd9de86411e8646d87eb7e74b79. Latest consensus state: current_term: 1 leader_uuid: "cc5efdd9de86411e8646d87eb7e74b79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5efdd9de86411e8646d87eb7e74b79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44027 } } }
I20260925 14:51:22.904165 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:22.904193 10675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:22.904779 10675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:22.906754 10675 catalog_manager.cc:1347] Generated new cluster ID: 57163d5437e7454bb488ebd0d0e1a1d4
I20260925 14:51:22.906806 10675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:51:22 dist-test-slave-gpc1 krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:22.913940 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:22.903643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38380 (local address 127.0.98.62:44027)
0925 14:51:22.903808 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:22.903819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:22.903921 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:51:22.904071 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:22.904079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:22.904139 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:22.905232 (+ 1093us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:22.905639 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:22.906955 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:22.907735 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:22.908205 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:22.908325 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:22.910664 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:22.910668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:22.910669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:22.910681 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:22.912363 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:22.912737 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:22.912740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:22.912740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:22.912839 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:22.913049 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:22.913050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:22.913050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:22.913287 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:22.913454 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:22.913536 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:51:22.913775 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260925 14:51:22.917919 10675 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:22.918642 10675 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:22.928692 10675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: Generated new TSK 0
I20260925 14:51:22.928920 10675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:51:22.961328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:23.041469 10691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:23.041710 10691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:23.041747 10691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:23.042889 10691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:23.042956 10691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:23.043002 10691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:23.044492 10691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:23.044761 10691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:23.044992 10691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:23.049913 10699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:23.049958 10691 server_base.cc:1034] running on GCE node
W20260925 14:51:23.049983 10696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:23.049922 10697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:23.050300 10691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:23.050621 10691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:23.051823 10691 hybrid_clock.cc:648] HybridClock initialized: now 1790347883051810 us; error 28 us; skew 500 ppm
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:23.054337 10691 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:23.054808 10691 webserver.cc:460] Webserver started at http://127.0.98.1:38439/ using document root <none> and password file <none>
I20260925 14:51:23.055058 10691 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:23.055130 10691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:23.055272 10691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:23.056344 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "c13bfdff7ecce57164e96c8f853d10aa"
server_key_iv: "3a24b409ca270242bbb3759efee3bba2"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.056710 10691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "c13bfdff7ecce57164e96c8f853d10aa"
server_key_iv: "3a24b409ca270242bbb3759efee3bba2"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.058049 10691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:23.058825 10706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.058984 10691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.059129 10691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "c13bfdff7ecce57164e96c8f853d10aa"
server_key_iv: "3a24b409ca270242bbb3759efee3bba2"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.059224 10691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:23.073834 10691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:23.075918 10691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:23.076093 10691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:23.076334 10691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:23.076635 10691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:23.076689 10691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.076731 10691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:23.076764 10691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.082532 10691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34809
I20260925 14:51:23.082607 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34809 every 8 connection(s)
I20260925 14:51:23.082939 10691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:23.086149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10691
I20260925 14:51:23.086257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:23.086503 392 external_mini_cluster.cc:1428] Setting key eb11d7d554e6cf5b4ec346a5af173a80
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:23.093470 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.083797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50123 (local address 127.0.98.62:44027)
0925 14:51:23.083879 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.083887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.084045 (+ 158us) server_negotiation.cc:407] Connection header received
0925 14:51:23.084516 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.084517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.084532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.084597 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:23.085495 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.086039 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.087520 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.087728 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.087805 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.090258 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:23.090260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:23.090260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:23.090268 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:23.092052 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.092460 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.092462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.092462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.092528 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.092761 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.092762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.092762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.092912 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:23.092996 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.093248 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.093347 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:51:23.093495 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.083724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44027 (local address 127.0.98.1:50123)
0925 14:51:23.083902 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.083943 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.084429 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.084655 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.084658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.085220 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:23.085410 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.085415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.086152 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.086153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.087422 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.087425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.087812 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.087813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.087954 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.088544 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:23.088557 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:23.090169 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:23.092145 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.092151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.092153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.092383 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.092597 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.092598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.092598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.092696 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.092989 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:23.092993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:23.093115 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.093196 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.093334 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 14:51:23.094005 10820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44027
I20260925 14:51:23.094112 10820 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:23.094314 10820 heartbeater.cc:511] Master 127.0.98.62:44027 requested a full tablet report, sending...
I20260925 14:51:23.094934 10629 ts_manager.cc:194] Registered new tserver with Master: 8f28a5c77d3b4ec0bbff8774d8e9ee00 (127.0.98.1:34809)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:51:23.095616 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:51:23.113878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:23.193779 10825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:23.193987 10825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:23.194010 10825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:23.195237 10825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:23.195286 10825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:23.195339 10825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:23.196799 10825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:23.197053 10825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:23.197286 10825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:23.202520 10833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:23.202548 10825 server_base.cc:1034] running on GCE node
W20260925 14:51:23.202520 10831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:23.202533 10830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:23.203001 10825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:23.203303 10825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:23.204453 10825 hybrid_clock.cc:648] HybridClock initialized: now 1790347883204437 us; error 34 us; skew 500 ppm
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:23.207250 10825 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:23.207828 10825 webserver.cc:460] Webserver started at http://127.0.98.2:46625/ using document root <none> and password file <none>
I20260925 14:51:23.208091 10825 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:23.208163 10825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:23.208320 10825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:23.209450 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "4b0388e783c947b084f10df602208046"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "a955d225d7f7d07d6b98fbeb7f148c89"
server_key_iv: "c3392fff0cd612b4988bdf35d3b73474"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.209837 10825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4b0388e783c947b084f10df602208046"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "a955d225d7f7d07d6b98fbeb7f148c89"
server_key_iv: "c3392fff0cd612b4988bdf35d3b73474"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.211423 10825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 14:51:23.212289 10840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.212538 10825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:23.212652 10825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "4b0388e783c947b084f10df602208046"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "a955d225d7f7d07d6b98fbeb7f148c89"
server_key_iv: "c3392fff0cd612b4988bdf35d3b73474"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.212733 10825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:23.230542 10825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:23.233163 10825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:23.233352 10825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:23.233613 10825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:23.233950 10825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:23.234016 10825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.234061 10825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:23.234110 10825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.240659 10825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34585
I20260925 14:51:23.240702 10953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34585 every 8 connection(s)
I20260925 14:51:23.241050 10825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:23.248739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10825
I20260925 14:51:23.248869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:23.249104 392 external_mini_cluster.cc:1428] Setting key 837ff80ffdddfa5741b2d1c1553ea6a3
I20260925 14:51:23.251256 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.241882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48487 (local address 127.0.98.62:44027)
0925 14:51:23.241964 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.241966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.242223 (+ 257us) server_negotiation.cc:407] Connection header received
0925 14:51:23.242779 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.242780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.242791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.242851 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:23.243899 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.244418 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.245484 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.245686 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.245755 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.247917 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:23.247918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:23.247919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:23.247925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:23.249657 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.250206 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.250207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.250207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.250269 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.250545 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.250546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.250546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.250697 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:23.250767 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.251013 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.251107 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:51:23.251291 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.241830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44027 (local address 127.0.98.2:48487)
0925 14:51:23.242088 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.242128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.242679 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.242913 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.242917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.243571 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:23.243789 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.243794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.244527 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.244528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.245393 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.245396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.245770 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.245771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.245872 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.246474 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:23.246482 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:23.247835 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:23.249843 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.249851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.249853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.250138 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.250364 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.250365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.250365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.250479 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.250761 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:23.250765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:23.250883 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.250962 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.251085 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260925 14:51:23.252162 10954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44027
I20260925 14:51:23.252280 10954 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:23.252480 10954 heartbeater.cc:511] Master 127.0.98.62:44027 requested a full tablet report, sending...
I20260925 14:51:23.252848 10629 ts_manager.cc:194] Registered new tserver with Master: 4b0388e783c947b084f10df602208046 (127.0.98.2:34585)
I20260925 14:51:23.253238 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48487
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:51:23.275835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:23.356487 10959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:23.356717 10959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:23.356755 10959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:23.357898 10959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:23.357966 10959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:23.358011 10959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:23.359467 10959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:51:23.359721 10959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:23.359953 10959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:23.365046 10965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:23.365063 10964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:23.365062 10967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:23.365126 10959 server_base.cc:1034] running on GCE node
I20260925 14:51:23.365389 10959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:23.365584 10959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:23.366717 10959 hybrid_clock.cc:648] HybridClock initialized: now 1790347883366704 us; error 28 us; skew 500 ppm
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:23.369176 10959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:23.369657 10959 webserver.cc:460] Webserver started at http://127.0.98.3:39295/ using document root <none> and password file <none>
I20260925 14:51:23.369897 10959 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:23.370004 10959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:23.370146 10959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:23.371510 10959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "02fece1002584d05bc9a883675514f81"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "f18e65ec6e94cb6480d6479d5599d17b"
server_key_iv: "00943acd7f4cff0a90e8ac92e83655dd"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.371903 10959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02fece1002584d05bc9a883675514f81"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "f18e65ec6e94cb6480d6479d5599d17b"
server_key_iv: "00943acd7f4cff0a90e8ac92e83655dd"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.373217 10959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:23.373978 10974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.374179 10959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:23.374274 10959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "02fece1002584d05bc9a883675514f81"
format_stamp: "Formatted at 2026-09-25 14:51:23 on dist-test-slave-gpc1"
server_key: "f18e65ec6e94cb6480d6479d5599d17b"
server_key_iv: "00943acd7f4cff0a90e8ac92e83655dd"
server_key_version: "encryptionkey@0"
I20260925 14:51:23.374382 10959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:23.386504 10959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:23.389083 10959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:23.389232 10959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:23.389492 10959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:23.389838 10959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:23.389889 10959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.389920 10959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:23.389943 10959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:23.396734 10959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35735
I20260925 14:51:23.396836 11087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35735 every 8 connection(s)
I20260925 14:51:23.397219 10959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:23.400472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 10959
I20260925 14:51:23.400573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:23.400761 392 external_mini_cluster.cc:1428] Setting key dba44fc644bee14eaafc6db77fb3fb51
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[10584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:23.408866 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.397939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54391 (local address 127.0.98.62:44027)
0925 14:51:23.397992 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.397995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.398282 (+ 287us) server_negotiation.cc:407] Connection header received
0925 14:51:23.398861 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.398862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.398873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.398997 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:23.400178 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.400972 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.402438 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.402652 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.402737 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.405522 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:23.405523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:23.405524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:23.405533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:23.407251 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.407741 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.407742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.407742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.407796 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.408177 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.408178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.408179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.408284 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:23.408364 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.408699 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.408777 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:51:23.409067 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.397951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44027 (local address 127.0.98.3:54391)
0925 14:51:23.398127 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.398178 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.398761 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.399087 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.399097 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.399851 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:23.400070 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.400074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.401081 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.401082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.402300 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.402303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.402746 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.402747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.402876 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.403450 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:23.403463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:23.405422 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:23.407344 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.407350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.407353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.407663 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.407868 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:23.407869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:23.407869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:23.408011 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:23.408369 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:23.408376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:23.408546 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.408672 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.408828 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260925 14:51:23.409662 11088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44027
I20260925 14:51:23.409780 11088 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:23.409993 11088 heartbeater.cc:511] Master 127.0.98.62:44027 requested a full tablet report, sending...
I20260925 14:51:23.410454 10629 ts_manager.cc:194] Registered new tserver with Master: 02fece1002584d05bc9a883675514f81 (127.0.98.3:35735)
I20260925 14:51:23.410831 10629 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54391
I20260925 14:51:23.412340 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:23.421626 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.413892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.0.98.62:44027)
0925 14:51:23.413958 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.413960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.413969 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:23.414032 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.414033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.414043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.414095 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:23.414921 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.415682 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.416735 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.417096 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.417172 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.418565 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:23.418567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:23.418568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:23.418575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:23.420286 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.420609 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.420610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.420610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.420672 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:23.421027 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:23.421028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:23.421029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:23.421225 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:23.421340 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.421431 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.421510 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:51:23.423431 10629 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:23.426101 10629 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:23.432909 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34585 (local address 127.0.0.1:42896)
0925 14:51:23.428364 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.428402 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.428470 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.428791 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.428794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.428796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:23.428863 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.428865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430292 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.430293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.431733 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.431735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.432621 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.432622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.432707 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.432721 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.432789 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.432836 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":140,"thread_start_us":63,"threads_started":1}
I20260925 14:51:23.433051 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35735 (local address 127.0.0.1:56858)
0925 14:51:23.428704 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.428714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.428944 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.429369 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.429370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.429372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:23.429410 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.429411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430637 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.430638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.431919 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.431922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.432773 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.432774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.432848 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.432853 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.432886 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.432956 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":35,"threads_started":1}
I20260925 14:51:23.433182 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.0.98.2:34585)
0925 14:51:23.428552 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.428556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.428567 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:23.428618 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.428619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.428670 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.428764 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:23.428930 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430197 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.431822 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.432540 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.432615 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.432674 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.433031 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.433087 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":112,"threads_started":1}
I20260925 14:51:23.433424 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.0.98.3:35735)
0925 14:51:23.429208 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.429213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.429224 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:23.429260 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.429262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.429277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.429356 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:23.429467 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430518 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.432020 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.432678 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.433157 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.433230 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.433268 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.433317 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":89,"threads_started":1}
I20260925 14:51:23.433841 10888 tablet_service.cc:1467] Processing CreateTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 (DEFAULT_TABLE table=test-table [id=065959c6941e40b1934b8e40128cd604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:23.434118 10888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:23.434134 11022 tablet_service.cc:1467] Processing CreateTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 (DEFAULT_TABLE table=test-table [id=065959c6941e40b1934b8e40128cd604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:23.434510 11022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:23.435186 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34809 (local address 127.0.0.1:51680)
0925 14:51:23.428970 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.428979 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.429024 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.429278 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.429278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.429280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:23.429316 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.429317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430237 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.430238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.432307 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.432309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.434953 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.434954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.435025 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.435028 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.435082 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.435120 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":1119,"thread_start_us":81,"threads_started":1}
I20260925 14:51:23.435549 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.428719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.98.1:34809)
0925 14:51:23.429065 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.429070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.429080 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:23.429116 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.429119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.429133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.429203 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:23.429415 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.430132 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.432387 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.434803 (+ 2416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.434869 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.434920 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.435425 (+ 505us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.435472 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":88,"threads_started":1}
I20260925 14:51:23.437103 11102 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Bootstrap starting.
I20260925 14:51:23.437677 11103 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Bootstrap starting.
I20260925 14:51:23.437816 11102 tablet_bootstrap.cc:654] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:23.438274 11103 tablet_bootstrap.cc:654] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:23.438385 11102 log.cc:824] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:23.438278 10754 tablet_service.cc:1467] Processing CreateTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 (DEFAULT_TABLE table=test-table [id=065959c6941e40b1934b8e40128cd604]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:23.438622 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:23.438906 11103 log.cc:824] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:23.439226 11102 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: No bootstrap required, opened a new log
I20260925 14:51:23.439303 11102 ts_tablet_manager.cc:1397] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:23.439633 11103 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: No bootstrap required, opened a new log
I20260925 14:51:23.439725 11103 ts_tablet_manager.cc:1397] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:51:23.441208 11103 raft_consensus.cc:348] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.441368 11103 raft_consensus.cc:374] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:23.441267 11102 raft_consensus.cc:348] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.441413 11103 raft_consensus.cc:729] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fece1002584d05bc9a883675514f81, State: Initialized, Role: FOLLOWER
I20260925 14:51:23.441442 11102 raft_consensus.cc:374] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:23.441488 11102 raft_consensus.cc:729] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0388e783c947b084f10df602208046, State: Initialized, Role: FOLLOWER
I20260925 14:51:23.441547 11103 consensus_queue.cc:260] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.441619 11102 consensus_queue.cc:260] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.441820 11103 ts_tablet_manager.cc:1428] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:51:23.441886 11102 ts_tablet_manager.cc:1428] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:23.441941 11088 heartbeater.cc:503] Master 127.0.98.62:44027 was elected leader, sending a full tablet report...
I20260925 14:51:23.441960 10954 heartbeater.cc:503] Master 127.0.98.62:44027 was elected leader, sending a full tablet report...
I20260925 14:51:23.442060 11106 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: Bootstrap starting.
I20260925 14:51:23.442935 11106 tablet_bootstrap.cc:654] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:23.443509 11106 log.cc:824] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:23.444149 11106 tablet_bootstrap.cc:492] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: No bootstrap required, opened a new log
I20260925 14:51:23.444201 11106 ts_tablet_manager.cc:1397] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:23.445472 11106 raft_consensus.cc:348] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.445604 11106 raft_consensus.cc:374] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:23.445626 11106 raft_consensus.cc:729] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f28a5c77d3b4ec0bbff8774d8e9ee00, State: Initialized, Role: FOLLOWER
I20260925 14:51:23.445734 11106 consensus_queue.cc:260] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.446002 11106 ts_tablet_manager.cc:1428] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:23.446038 10820 heartbeater.cc:503] Master 127.0.98.62:44027 was elected leader, sending a full tablet report...
I20260925 14:51:23.450837 11108 raft_consensus.cc:482] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:23.450927 11108 raft_consensus.cc:504] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.451234 11108 leader_election.cc:288] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fece1002584d05bc9a883675514f81 (127.0.98.3:35735), 8f28a5c77d3b4ec0bbff8774d8e9ee00 (127.0.98.1:34809)
I20260925 14:51:23.454195 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.451473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41755 (local address 127.0.98.3:35735)
0925 14:51:23.451544 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.451546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.451570 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:51:23.451592 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.451592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.451602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.451642 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:23.451869 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.452328 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.453446 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.453825 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.453917 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.453928 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.454090 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.454128 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:51:23.454195 10956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.451400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35735 (local address 127.0.98.2:41755)
0925 14:51:23.451464 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.451476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.451517 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.451667 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.451668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.451670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:23.451742 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.451744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.452420 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.452421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.453360 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.453362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.453918 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.453918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.453997 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.454007 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.454082 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.454127 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:51:23.454303 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.451537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34809 (local address 127.0.98.2:53477)
0925 14:51:23.451742 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.451754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:23.451799 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:23.451954 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:23.451954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:23.451956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:23.452043 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.452044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.452635 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.452635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.453494 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:23.453496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.454039 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.454040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:23.454115 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.454119 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:23.454189 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:51:23.454226 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":124,"threads_started":1}
I20260925 14:51:23.454293 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.451765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53477 (local address 127.0.98.1:34809)
0925 14:51:23.451805 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.451807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.451817 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:23.451882 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.451883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.451893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.451949 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:23.452109 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.452557 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.453588 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.453944 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.454028 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.454036 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.454197 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.454233 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:23.454541 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa5c4d0fcbb4e0ea3c98f448547e8e3" candidate_uuid: "4b0388e783c947b084f10df602208046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fece1002584d05bc9a883675514f81" is_pre_election: true
I20260925 14:51:23.454636 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa5c4d0fcbb4e0ea3c98f448547e8e3" candidate_uuid: "4b0388e783c947b084f10df602208046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" is_pre_election: true
I20260925 14:51:23.454677 11042 raft_consensus.cc:2454] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0388e783c947b084f10df602208046 in term 0.
I20260925 14:51:23.454751 10774 raft_consensus.cc:2454] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0388e783c947b084f10df602208046 in term 0.
I20260925 14:51:23.454953 10841 leader_election.cc:302] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0388e783c947b084f10df602208046, 8f28a5c77d3b4ec0bbff8774d8e9ee00; no voters:
I20260925 14:51:23.455104 11108 raft_consensus.cc:2781] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:23.455165 11108 raft_consensus.cc:482] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:23.455183 11108 raft_consensus.cc:3037] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:23.455873 11108 raft_consensus.cc:504] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.455994 11108 leader_election.cc:288] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [CANDIDATE]: Term 1 election: Requested vote from peers 02fece1002584d05bc9a883675514f81 (127.0.98.3:35735), 8f28a5c77d3b4ec0bbff8774d8e9ee00 (127.0.98.1:34809)
I20260925 14:51:23.456161 11042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa5c4d0fcbb4e0ea3c98f448547e8e3" candidate_uuid: "4b0388e783c947b084f10df602208046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fece1002584d05bc9a883675514f81"
I20260925 14:51:23.456156 10774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efa5c4d0fcbb4e0ea3c98f448547e8e3" candidate_uuid: "4b0388e783c947b084f10df602208046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00"
I20260925 14:51:23.456229 11042 raft_consensus.cc:3037] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:23.456250 10774 raft_consensus.cc:3037] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:23.456825 11042 raft_consensus.cc:2454] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0388e783c947b084f10df602208046 in term 1.
I20260925 14:51:23.456976 10841 leader_election.cc:302] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fece1002584d05bc9a883675514f81, 4b0388e783c947b084f10df602208046; no voters:
I20260925 14:51:23.457062 10774 raft_consensus.cc:2454] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0388e783c947b084f10df602208046 in term 1.
I20260925 14:51:23.457086 11108 raft_consensus.cc:2781] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:23.457244 11108 raft_consensus.cc:686] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0388e783c947b084f10df602208046, State: Running, Role: LEADER
I20260925 14:51:23.457340 11108 consensus_queue.cc:237] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } }
I20260925 14:51:23.458123 10628 catalog_manager.cc:5495] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0388e783c947b084f10df602208046 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4b0388e783c947b084f10df602208046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0388e783c947b084f10df602208046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 } health_report { overall_health: UNKNOWN } } }
I20260925 14:51:23.463177 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.461055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42908 (local address 127.0.98.2:34585)
0925 14:51:23.461110 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:23.461112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:23.461118 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:51:23.461154 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:23.461155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:23.461169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:23.461217 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:23.461363 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.461784 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.462542 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:23.462714 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:23.462767 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:23.462879 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:23.462965 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:23.463022 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:23.463070 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:51:23.463086 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:51:23.464708 10774 raft_consensus.cc:1261] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 1 FOLLOWER]: Refusing update from remote peer 4b0388e783c947b084f10df602208046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:23.464726 11042 raft_consensus.cc:1261] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 1 FOLLOWER]: Refusing update from remote peer 4b0388e783c947b084f10df602208046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:23.464993 11108 consensus_queue.cc:1035] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f28a5c77d3b4ec0bbff8774d8e9ee00" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:23.465112 11112 consensus_queue.cc:1035] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fece1002584d05bc9a883675514f81" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:23.467614 11115 mvcc.cc:205] Tried to move back new op lower bound from 7333264930669244416 to 7333264930642006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333264930669244416}]
I20260925 14:51:23.468091 11116 mvcc.cc:205] Tried to move back new op lower bound from 7333264930669244416 to 7333264930642006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333264930669244416}]
I20260925 14:51:23.468603 11118 mvcc.cc:205] Tried to move back new op lower bound from 7333264930669244416 to 7333264930642006016. Current Snapshot: MvccSnapshot[applied={T|T < 7333264930669244416}]
I20260925 14:51:23.470201 10628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:23.470294 10628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:23.471035 10628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc5efdd9de86411e8646d87eb7e74b79: Sending DeleteTablet for 3 replicas of tablet efa5c4d0fcbb4e0ea3c98f448547e8e3
I20260925 14:51:23.471328 10888 tablet_service.cc:1514] Processing DeleteTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42896
I20260925 14:51:23.471369 11022 tablet_service.cc:1514] Processing DeleteTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56858
I20260925 14:51:23.471580 10754 tablet_service.cc:1514] Processing DeleteTablet for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51680
I20260925 14:51:23.471786 11126 tablet_replica.cc:330] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00: stopping tablet replica
I20260925 14:51:23.471800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10691
I20260925 14:51:23.471940 11126 raft_consensus.cc:2229] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:23.472092 11126 raft_consensus.cc:2258] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 8f28a5c77d3b4ec0bbff8774d8e9ee00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:23.474866 11124 tablet_replica.cc:330] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: stopping tablet replica
I20260925 14:51:23.474982 11124 raft_consensus.cc:2229] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:23.475122 11124 raft_consensus.cc:2258] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:23.475485 11124 ts_tablet_manager.cc:1905] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:23.477154 11124 ts_tablet_manager.cc:1918] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:23.477238 11124 log.cc:1196] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/efa5c4d0fcbb4e0ea3c98f448547e8e3
I20260925 14:51:23.477504 11124 ts_tablet_manager.cc:1939] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 4b0388e783c947b084f10df602208046: Deleting consensus metadata
I20260925 14:51:23.478727 10613 catalog_manager.cc:4841] TS 4b0388e783c947b084f10df602208046 (127.0.98.2:34585): tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 (table test-table [id=065959c6941e40b1934b8e40128cd604]) successfully deleted
W20260925 14:51:23.479096 10613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34809 (error 108)
I20260925 14:51:23.479566 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:23.479443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34809 (local address 127.0.0.1:51686)
0925 14:51:23.479494 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:23.479530 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:51:23.479629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10825
W20260925 14:51:23.479758 10613 catalog_manager.cc:4584] TS 8f28a5c77d3b4ec0bbff8774d8e9ee00 (127.0.98.1:34809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efa5c4d0fcbb4e0ea3c98f448547e8e3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34809: connect: Connection refused (error 111)
I20260925 14:51:23.480235 11125 tablet_replica.cc:330] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: stopping tablet replica
I20260925 14:51:23.480348 11125 raft_consensus.cc:2229] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:23.480453 11125 raft_consensus.cc:2258] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:23.480859 11125 ts_tablet_manager.cc:1905] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:23.482138 11125 ts_tablet_manager.cc:1918] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:23.482227 11125 log.cc:1196] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/efa5c4d0fcbb4e0ea3c98f448547e8e3
I20260925 14:51:23.482533 11125 ts_tablet_manager.cc:1939] T efa5c4d0fcbb4e0ea3c98f448547e8e3 P 02fece1002584d05bc9a883675514f81: Deleting consensus metadata
I20260925 14:51:23.482900 10613 catalog_manager.cc:4841] TS 02fece1002584d05bc9a883675514f81 (127.0.98.3:35735): tablet efa5c4d0fcbb4e0ea3c98f448547e8e3 (table test-table [id=065959c6941e40b1934b8e40128cd604]) successfully deleted
I20260925 14:51:23.485574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10959
I20260925 14:51:23.490051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 10597
2026-09-25T14:51:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1924 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[11130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[11130](info): set up 2 sockets
Sep 25 14:51:23 dist-test-slave-gpc1 krb5kdc[11130](info): commencing operation
krb5kdc: starting...
W20260925 14:51:24.527613 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:24 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 14:51:24.693421 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbEnjDj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/postgres -l logfile start
2026-09-25 14:51:27.065 UTC [11160] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:51:27.066 UTC [11160] LOG: listening on IPv4 address "127.0.98.20", port 50275
2026-09-25 14:51:27.069 UTC [11160] LOG: listening on Unix socket "/tmp/.s.PGSQL.50275"
2026-09-25 14:51:27.091 UTC [11163] LOG: database system was shut down at 2026-09-25 14:51:25 UTC
2026-09-25 14:51:27.094 UTC [11160] LOG: database system is ready to accept connections
I20260925 14:51:28.043226 392 mini_postgres.cc:96] Postgres bound to 50275
2026-09-25 14:51:28.049 UTC [11171] FATAL: database "slave" does not exist
127.0.98.20:50275 - accepting connections
I20260925 14:51:28.049901 392 mini_ranger.cc:161] Starting Ranger...
I20260925 14:51:28.064344 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:51:28.536813 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:51:28.536880 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin
2026-09-25 14:51:28,820 [I] DB FLAVOR :POSTGRES
2026-09-25 14:51:28,821 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:51:28,821 [I] Checking connection..
2026-09-25 14:51:28,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:51:29,065 [I] Checking connection passed.
2026-09-25 14:51:29,065 [I] --------- Verifying version history table ---------
2026-09-25 14:51:29,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:51:29,291 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:51:29,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:51:29,548 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:51:29,548 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:51:29,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:51:29.772 UTC [11235] WARNING: there is no transaction in progress
2026-09-25 14:51:29,779 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:51:29,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:51:30,019 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:51:30,019 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:51:30,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:51:30,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:51:30,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gpc1.c.gcp-upstream.internal','N') ;"
2026-09-25 14:51:30,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:51:30,992 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:51:30,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:51:31,221 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:51:31,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:51:31,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:51:31,729 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:51:31,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:51:32.700 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.716 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.733 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.749 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.766 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.784 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.839 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.845 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.859 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.869 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.879 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.893 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.902 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.908 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.915 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:32.921 UTC [11343] WARNING: there is no transaction in progress
2026-09-25 14:51:33,533 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:51:33,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:51:33,754 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:51:33,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:51:33,999 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:51:34,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:51:34,242 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:51:34,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:51:34,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:51:34,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gpc1.c.gcp-upstream.internal';"
2026-09-25 14:51:34,952 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:51:34,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:51:35,201 [I] DB_PATCHES have already been applied
I20260925 14:51:35.206915 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbEnjDj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:51:35.206981 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 14:51:35.209884 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:52821
Sep 25, 2026 2:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:51:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:51:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:51:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:51:36.314106 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.264s user 0.000s sys 0.009s
Sep 25, 2026 2:51:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:51:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 2:51:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:51:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:51:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:51:58.119410 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:51:58 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:51:58.219125 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:51:58.228902 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:51:59.121908 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:51:59.122020 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms
I20260925 14:51:59.122038 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:50275
2026-09-25 14:51:59,301 [I] DB FLAVOR :POSTGRES
2026-09-25 14:51:59,301 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:51:59,301 [I] Checking connection
2026-09-25 14:51:59,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:51:59,587 [I] connection success
2026-09-25 14:51:59,587 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:51:59,587 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:51:59,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:51:59,851 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:51:59,852 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:51:59,852 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:51:59,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:50275/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:52:00,112 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:52:00.605221 392 mini_ranger_kms.cc:310] Created kms service
I20260925 14:52:00.808758 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:52:00.855371 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:52:00 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:01.247493 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:52:01.247617 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:52:01.247639 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 14:52:01.250937 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11523
I20260925 14:52:01.251096 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:36951
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:52:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 14:52:03 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:03.491199 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.272s user 0.006s sys 0.152s
I20260925 14:52:03.491302 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:52:03.491375 392 mini_ranger_kms.cc:401] 127.0.98.20:36951/kms/v1/keys
Sep 25 14:52:03 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:52:04 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347923, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:52:06 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:06.492969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37287
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36951/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:06.574097 11574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:06.574312 11574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:06.574335 11574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:06.575547 11574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:06.575590 11574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:06.575604 11574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:06.575616 11574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:06.575629 11574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:06.577124 11574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:36951/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37287
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:06.577325 11574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:06.577491 11574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:52:06.582780 11574 server_base.cc:1034] running on GCE node
W20260925 14:52:06.582749 11580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:06.582749 11582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:06.582741 11579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:06.583249 11574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:06.583570 11574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:06.584751 11574 hybrid_clock.cc:648] HybridClock initialized: now 1790347926584738 us; error 30 us; skew 500 ppm
Sep 25 14:52:06 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:06.587255 11574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:06.587658 11574 webserver.cc:460] Webserver started at http://127.0.98.62:42677/ using document root <none> and password file <none>
I20260925 14:52:06.587839 11574 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:06.587873 11574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:06.587961 11574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:06 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:06.636726 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "41132e9224504096a6eeb7d2f2827c39"
format_stamp: "Formatted at 2026-09-25 14:52:06 on dist-test-slave-gpc1"
server_key: "876ca71f5a00d57f6bd2a71cab73255a"
server_key_iv: "f88d9ba34b0a0b040be88cfbc8666f21"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:06.637181 11574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "41132e9224504096a6eeb7d2f2827c39"
format_stamp: "Formatted at 2026-09-25 14:52:06 on dist-test-slave-gpc1"
server_key: "876ca71f5a00d57f6bd2a71cab73255a"
server_key_iv: "f88d9ba34b0a0b040be88cfbc8666f21"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:06.639117 11574 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:52:06.654958 11589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:06.655174 11574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:06.655280 11574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "41132e9224504096a6eeb7d2f2827c39"
format_stamp: "Formatted at 2026-09-25 14:52:06 on dist-test-slave-gpc1"
server_key: "876ca71f5a00d57f6bd2a71cab73255a"
server_key_iv: "f88d9ba34b0a0b040be88cfbc8666f21"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:06.655375 11574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:06.667258 11574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:06.669625 11574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:06.669795 11574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:06.675272 11574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37287
I20260925 14:52:06.675302 11641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37287 every 8 connection(s)
I20260925 14:52:06.675736 11574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:06.676465 11642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:06.678661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 11574
I20260925 14:52:06.678757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:06.678929 392 external_mini_cluster.cc:1428] Setting key ad468d35702aff5541f88d3681590f70
I20260925 14:52:06.678920 11642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: Bootstrap starting.
I20260925 14:52:06.679721 11642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:06.680083 11642 log.cc:824] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:06.680898 11642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: No bootstrap required, opened a new log
I20260925 14:52:06.682991 11642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260925 14:52:06.683156 11642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:06.683187 11642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41132e9224504096a6eeb7d2f2827c39, State: Initialized, Role: FOLLOWER
I20260925 14:52:06.683346 11642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260925 14:52:06.683426 11642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:06.683496 11642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:06.683652 11642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:06.684374 11642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260925 14:52:06.684536 11642 leader_election.cc:302] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41132e9224504096a6eeb7d2f2827c39; no voters:
I20260925 14:52:06.684731 11642 leader_election.cc:288] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:06.684777 11647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:06.685038 11647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [term 1 LEADER]: Becoming Leader. State: Replica: 41132e9224504096a6eeb7d2f2827c39, State: Running, Role: LEADER
I20260925 14:52:06.685156 11647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260925 14:52:06.685194 11642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:06.685521 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41132e9224504096a6eeb7d2f2827c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260925 14:52:06.685621 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:06.685801 11647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41132e9224504096a6eeb7d2f2827c39. Latest consensus state: current_term: 1 leader_uuid: "41132e9224504096a6eeb7d2f2827c39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41132e9224504096a6eeb7d2f2827c39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260925 14:52:06.685873 11647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:06.687059 11642 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 14:52:06 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:06.690934 11651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:06.691823 11651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:06.697080 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:06.679611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.0.98.62:37287)
0925 14:52:06.679791 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:06.679801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:06.679836 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:52:06.680454 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:06.680461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:06.680520 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:06.680702 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:06.681443 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:06.683372 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:06.684612 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:06.685540 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:06.686187 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:06.689683 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:06.689744 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:06.689744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:06.689859 (+ 115us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:06.694637 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:06.695116 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:06.695119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:06.695119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:06.695220 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:06.695443 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:06.695444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:06.695445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:06.695790 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:06.696429 (+ 639us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:06.696575 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:52:06.696765 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260925 14:52:06.698863 11651 catalog_manager.cc:1347] Generated new cluster ID: 41fdfb68c372428aa203a89a1b1b3296
I20260925 14:52:06.698982 11651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:06.731829 11651 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:06.732808 11651 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:06.739203 11651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: Generated new TSK 0
I20260925 14:52:06.740574 11651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:52:07.429021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:07.517735 11687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:07.517980 11687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:07.518019 11687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:07.519276 11687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:07.519342 11687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:07.519388 11687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:07.520843 11687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:36951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:07.521100 11687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:07.521341 11687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:07.526335 11692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.526368 11695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.526526 11693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:07.526515 11687 server_base.cc:1034] running on GCE node
I20260925 14:52:07.526930 11687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:07.527194 11687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:07.528375 11687 hybrid_clock.cc:648] HybridClock initialized: now 1790347927528356 us; error 32 us; skew 500 ppm
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:07.531796 11687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:07.532297 11687 webserver.cc:460] Webserver started at http://127.0.98.1:36695/ using document root <none> and password file <none>
I20260925 14:52:07.532521 11687 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:07.532590 11687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:07.532752 11687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:07.546438 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "d510d202fe97483c8906e02b796b7e65"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "86a7f80604b32cccf241fbe6a3e17173"
server_key_iv: "e6ead7b878a98a48d1210477b42f1236"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.547025 11687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d510d202fe97483c8906e02b796b7e65"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "86a7f80604b32cccf241fbe6a3e17173"
server_key_iv: "e6ead7b878a98a48d1210477b42f1236"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.550761 11687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:07.566527 11702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.566748 11687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:07.566848 11687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "d510d202fe97483c8906e02b796b7e65"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "86a7f80604b32cccf241fbe6a3e17173"
server_key_iv: "e6ead7b878a98a48d1210477b42f1236"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.566977 11687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:07.592868 11687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:07.595388 11687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:07.595814 11687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:07.596151 11687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:07.596577 11687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:07.596678 11687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.596773 11687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:07.596845 11687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.604176 11687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34811
I20260925 14:52:07.604602 11687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:07.605248 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34811 every 8 connection(s)
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:07.615053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 11687
I20260925 14:52:07.615176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:07.615382 392 external_mini_cluster.cc:1428] Setting key ac8dd22c2e9906e6d86bd1cc89cb5b59
I20260925 14:52:07.617193 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:07.605380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.1:57419)
0925 14:52:07.605526 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:07.605575 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:07.606070 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:07.606695 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:07.606700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:07.606961 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:07.607264 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:07.607271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.608481 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.608483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:07.609896 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:07.609899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.610431 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.610433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:07.610594 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:07.611226 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:07.611237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:07.612887 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:07.615082 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:07.615090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:07.615093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:07.615390 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:07.615623 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:07.615624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:07.615624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:07.615752 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:07.616354 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:07.616361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:07.616608 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 14:52:07.616733 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 14:52:07.616912 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260925 14:52:07.618213 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:07.605390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57419 (local address 127.0.98.62:37287)
0925 14:52:07.606450 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:07.606451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:07.606459 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:07.606480 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:07.606481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:07.606491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:07.607391 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:07.607439 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.608352 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.609993 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.610286 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.610848 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:07.613005 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:07.613006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:07.613006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:07.613015 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:07.614968 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.615483 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.615484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.615484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.615549 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.616113 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.616114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.616115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.616252 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:07.616336 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:07.617977 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0925 14:52:07.618078 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1033,"thread_start_us":101,"threads_started":1}
I20260925 14:52:07.618880 11816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260925 14:52:07.618988 11816 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:07.619177 11816 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260925 14:52:07.619736 11606 ts_manager.cc:194] Registered new tserver with Master: d510d202fe97483c8906e02b796b7e65 (127.0.98.1:34811)
I20260925 14:52:07.620358 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57419
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:52:07.650439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:07.766757 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:07.766991 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:07.767031 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:07.768227 11824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:07.768312 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:07.768357 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:07.769860 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:36951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:07.770125 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:07.770426 11824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:07.775359 11834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.775529 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:07.775421 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:07.776077 11824 server_base.cc:1034] running on GCE node
I20260925 14:52:07.776307 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:07.776520 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:07.777658 11824 hybrid_clock.cc:648] HybridClock initialized: now 1790347927777627 us; error 49 us; skew 500 ppm
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:07.780290 11824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:07.780748 11824 webserver.cc:460] Webserver started at http://127.0.98.2:33343/ using document root <none> and password file <none>
I20260925 14:52:07.780989 11824 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:07.781042 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:07.781170 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:07.792516 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "5c7a0cb13d334d2788c3d1de530bc553"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "9473e0278ead4e0a59ca6070201ced94"
server_key_iv: "689ab46205ef0d713e00906a6cfe2a9d"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.793085 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7a0cb13d334d2788c3d1de530bc553"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "9473e0278ead4e0a59ca6070201ced94"
server_key_iv: "689ab46205ef0d713e00906a6cfe2a9d"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.798828 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260925 14:52:07.811750 11841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:07.811952 11824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:07.812083 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "5c7a0cb13d334d2788c3d1de530bc553"
format_stamp: "Formatted at 2026-09-25 14:52:07 on dist-test-slave-gpc1"
server_key: "9473e0278ead4e0a59ca6070201ced94"
server_key_iv: "689ab46205ef0d713e00906a6cfe2a9d"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:07.812209 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:07.847636 11824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:07.856608 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:07.856855 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:07.857170 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:07.857611 11824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:07.857707 11824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.857779 11824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:07.857836 11824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:07.867800 11824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33015
I20260925 14:52:07.868185 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:07.868759 11954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33015 every 8 connection(s)
I20260925 14:52:07.869479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 11824
I20260925 14:52:07.869608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:07.869797 392 external_mini_cluster.cc:1428] Setting key be59ca0da487642073e04a5a0a36c7be
Sep 25 14:52:07 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:52:07.881100 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:07.869036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45935 (local address 127.0.98.62:37287)
0925 14:52:07.869117 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:07.869119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:07.869380 (+ 261us) server_negotiation.cc:407] Connection header received
0925 14:52:07.870166 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:07.870167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:07.870179 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:07.870211 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:07.872202 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.872993 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.874661 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.874996 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.875068 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:07.877738 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:07.877739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:07.877740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:07.877748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:07.879579 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.880013 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.880014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.880014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.880082 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:07.880387 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:07.880388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:07.880389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:07.880537 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:07.880618 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:07.880898 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:52:07.881002 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:52:07.881297 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:07.869039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.2:45935)
0925 14:52:07.869212 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:07.869251 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:07.870041 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:07.870455 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:07.870460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:07.871746 (+ 1286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:07.872075 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:07.872083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.873129 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.873130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:07.874545 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:07.874548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:07.875099 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:07.875100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:07.875269 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:07.875920 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:07.875931 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:07.877622 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:07.879674 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:07.879681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:07.879685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:07.879929 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:07.880156 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:07.880156 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:07.880157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:07.880334 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:07.880612 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:07.880616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:07.880757 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:52:07.880846 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:52:07.880971 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 14:52:07.881945 11955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260925 14:52:07.882071 11955 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:07.882272 11955 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260925 14:52:07.882844 11606 ts_manager.cc:194] Registered new tserver with Master: 5c7a0cb13d334d2788c3d1de530bc553 (127.0.98.2:33015)
I20260925 14:52:07.883255 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:52:07.897420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:36951/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:08.018611 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:08.018967 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:08.019053 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:08.020896 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:08.021019 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:08.021102 11961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:08.023393 11961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:36951/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:08.023882 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:08.024206 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:08.030609 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.030830 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:08.030951 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:08.032073 11961 server_base.cc:1034] running on GCE node
I20260925 14:52:08.032395 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:08.032699 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:08.033870 11961 hybrid_clock.cc:648] HybridClock initialized: now 1790347928033838 us; error 45 us; skew 500 ppm
Sep 25 14:52:08 dist-test-slave-gpc1 krb5kdc[11130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:08.037503 11961 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:08.038101 11961 webserver.cc:460] Webserver started at http://127.0.98.3:33383/ using document root <none> and password file <none>
I20260925 14:52:08.038424 11961 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:08.038499 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:08.038648 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:08 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:08.060163 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dd62f7e4fbe4a4c8305988340469f93"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-gpc1"
server_key: "9bfc0e125e819c68cac6bbd8c05a11ad"
server_key_iv: "79e9271a4e05eed0f64149e116b579a5"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:08.060623 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd62f7e4fbe4a4c8305988340469f93"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-gpc1"
server_key: "9bfc0e125e819c68cac6bbd8c05a11ad"
server_key_iv: "79e9271a4e05eed0f64149e116b579a5"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:08.062829 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 14:52:08.078491 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.078750 11961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:08.078877 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "6dd62f7e4fbe4a4c8305988340469f93"
format_stamp: "Formatted at 2026-09-25 14:52:08 on dist-test-slave-gpc1"
server_key: "9bfc0e125e819c68cac6bbd8c05a11ad"
server_key_iv: "79e9271a4e05eed0f64149e116b579a5"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:08.078979 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:08.104554 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:08.107292 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:08.107534 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:08.107867 11961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:08.108315 11961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:08.108407 11961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.108530 11961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:08.108609 11961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:08.116940 11961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40979
I20260925 14:52:08.117583 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:08.117578 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40979 every 8 connection(s)
I20260925 14:52:08.121483 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 11961
I20260925 14:52:08.121626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:08.121848 392 external_mini_cluster.cc:1428] Setting key b1d6243874abb642e0ec91f2ea703b87
Sep 25 14:52:08 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:08.138758 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.118568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32831 (local address 127.0.98.62:37287)
0925 14:52:08.118613 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.118616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.127586 (+ 8970us) server_negotiation.cc:407] Connection header received
0925 14:52:08.128412 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.128414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.128428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.128460 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:08.129356 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.130158 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.131776 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.132097 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.132148 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.134982 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:08.134983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:08.134983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:08.134992 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:08.136839 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.137382 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.137384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.137384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.137452 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.137796 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.137797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.137798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.137948 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:08.138007 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.138541 (+ 534us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.138654 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:52:08.139593 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.118578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.3:32831)
0925 14:52:08.127382 (+ 8804us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.127471 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.128262 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.128674 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.128679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.128946 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:08.129246 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.129252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.130284 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.130285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.131663 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.131666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.132251 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.132252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.132409 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.133137 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:08.133148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:08.134875 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:08.136939 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.136947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.136950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.137280 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.137530 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:08.137531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:08.137531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:08.137702 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:08.138055 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:08.138061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:08.138254 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.139159 (+ 905us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.139345 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8737,"thread_start_us":84,"threads_started":1}
I20260925 14:52:08.140422 12090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260925 14:52:08.140537 12090 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:08.140745 12090 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260925 14:52:08.141191 11606 ts_manager.cc:194] Registered new tserver with Master: 6dd62f7e4fbe4a4c8305988340469f93 (127.0.98.3:40979)
I20260925 14:52:08.141746 11606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32831
I20260925 14:52:08.144409 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:08.154560 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.146197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53976 (local address 127.0.98.62:37287)
0925 14:52:08.146250 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.146252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.146419 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:52:08.146487 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.146488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.146499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.146570 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:08.147338 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.148120 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.149254 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.149544 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.149624 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.151308 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:08.151309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:08.151310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:08.151318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:08.153195 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.153607 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.153608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.153608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.153675 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:08.153909 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:08.153910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:08.153910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:08.154063 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:08.154171 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.154372 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.154462 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:52:08.156708 11606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:08.159134 11606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:08.168522 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.162252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34811 (local address 127.0.0.1:34248)
0925 14:52:08.162950 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.162959 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.162995 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.163550 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.163553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.163555 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:08.163625 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.163627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.165359 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.165360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.166762 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.166765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.168188 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.168189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.168285 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.168300 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.168371 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.168424 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":78,"threads_started":1}
I20260925 14:52:08.168960 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.162527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.98.1:34811)
0925 14:52:08.162669 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.162674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.163060 (+ 386us) server_negotiation.cc:407] Connection header received
0925 14:52:08.163100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.163103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.163120 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.163201 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:08.163705 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.164808 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.166863 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.168008 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.168082 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.168146 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.168846 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.168888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":55,"threads_started":1}
I20260925 14:52:08.170059 11750 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed (DEFAULT_TABLE table=test-table [id=c9e1bdeebf684fdf8ef55c18d79d8049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:08.170502 11750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca7cbfb01c4e73b4108c969dd528ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:08.173254 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.162239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.0.1:46050)
0925 14:52:08.162737 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.162781 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.162850 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.164910 (+ 2060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.164911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.164913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:08.164961 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.164962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.166283 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.166284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.168867 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.168870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.172985 (+ 4115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.172988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.173070 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.173075 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.173136 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.173179 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":407,"threads_started":1}
I20260925 14:52:08.173600 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.162519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.0.98.2:33015)
0925 14:52:08.163333 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.163338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.163346 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:08.163381 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.163383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.163397 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.163465 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:08.165051 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.166184 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.169904 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.172255 (+ 2351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.172334 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.172412 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.173475 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.173521 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":81,"threads_started":1}
I20260925 14:52:08.174234 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.162765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40979 (local address 127.0.0.1:33974)
0925 14:52:08.169594 (+ 6829us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.169606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.169648 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.169995 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.169996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.169998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:08.170031 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.170032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.171730 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.171731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.172730 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.172732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.174000 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.174001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.174067 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.174071 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.174129 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.174167 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6817,"thread_start_us":78,"threads_started":1}
I20260925 14:52:08.174274 11876 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed (DEFAULT_TABLE table=test-table [id=c9e1bdeebf684fdf8ef55c18d79d8049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:08.174666 11876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca7cbfb01c4e73b4108c969dd528ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:08.176060 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.163010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.3:40979)
0925 14:52:08.169820 (+ 6810us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.169825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.169835 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:08.169875 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.169878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.169895 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.170107 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:08.170555 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.171628 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.172847 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.173915 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.173980 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.174048 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.175916 (+ 1868us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.175967 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6764,"thread_start_us":74,"threads_started":1}
I20260925 14:52:08.176692 12024 tablet_service.cc:1467] Processing CreateTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed (DEFAULT_TABLE table=test-table [id=c9e1bdeebf684fdf8ef55c18d79d8049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:08.177009 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fca7cbfb01c4e73b4108c969dd528ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:08.179221 12109 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: Bootstrap starting.
I20260925 14:52:08.180312 12109 tablet_bootstrap.cc:654] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:08.180606 12110 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: Bootstrap starting.
I20260925 14:52:08.181339 12110 tablet_bootstrap.cc:654] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:08.181672 12110 log.cc:824] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:08.182184 12109 log.cc:824] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:08.182535 12111 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Bootstrap starting.
I20260925 14:52:08.183673 12110 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: No bootstrap required, opened a new log
I20260925 14:52:08.183758 12110 ts_tablet_manager.cc:1397] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:08.187131 12110 raft_consensus.cc:348] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.187336 12110 raft_consensus.cc:374] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:08.187387 12110 raft_consensus.cc:729] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d510d202fe97483c8906e02b796b7e65, State: Initialized, Role: FOLLOWER
I20260925 14:52:08.188072 12110 consensus_queue.cc:260] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.188198 12111 tablet_bootstrap.cc:654] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:08.188434 12110 ts_tablet_manager.cc:1428] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:52:08.188612 11816 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260925 14:52:08.190302 12109 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: No bootstrap required, opened a new log
I20260925 14:52:08.190428 12109 ts_tablet_manager.cc:1397] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260925 14:52:08.192010 12109 raft_consensus.cc:348] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.192176 12109 raft_consensus.cc:374] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:08.192219 12109 raft_consensus.cc:729] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7a0cb13d334d2788c3d1de530bc553, State: Initialized, Role: FOLLOWER
I20260925 14:52:08.192353 12109 consensus_queue.cc:260] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.192720 11955 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260925 14:52:08.192720 12109 ts_tablet_manager.cc:1428] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:08.193048 12111 log.cc:824] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:08.194391 12111 tablet_bootstrap.cc:492] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: No bootstrap required, opened a new log
I20260925 14:52:08.194468 12111 ts_tablet_manager.cc:1397] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260925 14:52:08.196552 12111 raft_consensus.cc:348] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.196731 12111 raft_consensus.cc:374] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:08.196774 12111 raft_consensus.cc:729] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd62f7e4fbe4a4c8305988340469f93, State: Initialized, Role: FOLLOWER
I20260925 14:52:08.196902 12111 consensus_queue.cc:260] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.197189 12090 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260925 14:52:08.197180 12111 ts_tablet_manager.cc:1428] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
Sep 25 14:52:08 dist-test-slave-gpc1 krb5kdc[11130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:08.271318 12117 raft_consensus.cc:482] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:08.271459 12117 raft_consensus.cc:504] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.271889 12117 leader_election.cc:288] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c7a0cb13d334d2788c3d1de530bc553 (127.0.98.2:33015), d510d202fe97483c8906e02b796b7e65 (127.0.98.1:34811)
I20260925 14:52:08.275898 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.272031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.98.3:44565)
0925 14:52:08.272250 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.272262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.272303 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.272575 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.272575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.272577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:08.272641 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.272643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.273611 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.273612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.274894 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.274897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.275658 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.275658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.275732 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.275736 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.275779 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.275820 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260925 14:52:08.276314 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.272448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47841 (local address 127.0.98.1:34811)
0925 14:52:08.272491 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.272493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.272951 (+ 458us) server_negotiation.cc:407] Connection header received
0925 14:52:08.273048 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.273049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.273060 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.273090 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:08.273300 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.273944 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.275355 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.275938 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.275984 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.275993 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.276205 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.276245 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:52:08.276398 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.272156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44565 (local address 127.0.98.2:33015)
0925 14:52:08.272422 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.272424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.272432 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:08.272460 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.272461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.272473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.272516 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:08.272714 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.273495 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.275013 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.275555 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.275618 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.275627 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.276293 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.276335 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260925 14:52:08.276495 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.272089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34811 (local address 127.0.98.3:47841)
0925 14:52:08.272895 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.272906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.272976 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.273152 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:08.273153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:08.273154 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:08.273229 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.273231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.274042 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.274042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.275249 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:08.275251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.276065 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.276066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:08.276131 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.276134 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:08.276387 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 14:52:08.276425 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":15,"thread_start_us":115,"threads_started":1}
I20260925 14:52:08.276687 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca7cbfb01c4e73b4108c969dd528ed" candidate_uuid: "6dd62f7e4fbe4a4c8305988340469f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7a0cb13d334d2788c3d1de530bc553" is_pre_election: true
I20260925 14:52:08.276839 11909 raft_consensus.cc:2454] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd62f7e4fbe4a4c8305988340469f93 in term 0.
I20260925 14:52:08.277148 11770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca7cbfb01c4e73b4108c969dd528ed" candidate_uuid: "6dd62f7e4fbe4a4c8305988340469f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d510d202fe97483c8906e02b796b7e65" is_pre_election: true
I20260925 14:52:08.277287 11770 raft_consensus.cc:2454] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd62f7e4fbe4a4c8305988340469f93 in term 0.
I20260925 14:52:08.277575 11978 leader_election.cc:302] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd62f7e4fbe4a4c8305988340469f93, d510d202fe97483c8906e02b796b7e65; no voters:
I20260925 14:52:08.277716 12117 raft_consensus.cc:2781] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:08.277781 12117 raft_consensus.cc:482] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:08.277802 12117 raft_consensus.cc:3037] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:08.279150 12117 raft_consensus.cc:504] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.279294 12117 leader_election.cc:288] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [CANDIDATE]: Term 1 election: Requested vote from peers 5c7a0cb13d334d2788c3d1de530bc553 (127.0.98.2:33015), d510d202fe97483c8906e02b796b7e65 (127.0.98.1:34811)
I20260925 14:52:08.279557 11909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca7cbfb01c4e73b4108c969dd528ed" candidate_uuid: "6dd62f7e4fbe4a4c8305988340469f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c7a0cb13d334d2788c3d1de530bc553"
I20260925 14:52:08.279649 11909 raft_consensus.cc:3037] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:08.280503 11909 raft_consensus.cc:2454] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd62f7e4fbe4a4c8305988340469f93 in term 1.
I20260925 14:52:08.280714 11980 leader_election.cc:302] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7a0cb13d334d2788c3d1de530bc553, 6dd62f7e4fbe4a4c8305988340469f93; no voters:
I20260925 14:52:08.280714 11770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fca7cbfb01c4e73b4108c969dd528ed" candidate_uuid: "6dd62f7e4fbe4a4c8305988340469f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d510d202fe97483c8906e02b796b7e65"
I20260925 14:52:08.280809 12117 raft_consensus.cc:2781] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:08.280792 11770 raft_consensus.cc:3037] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:08.280959 12117 raft_consensus.cc:686] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 1 LEADER]: Becoming Leader. State: Replica: 6dd62f7e4fbe4a4c8305988340469f93, State: Running, Role: LEADER
I20260925 14:52:08.281059 12117 consensus_queue.cc:237] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } }
I20260925 14:52:08.281612 11770 raft_consensus.cc:2454] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd62f7e4fbe4a4c8305988340469f93 in term 1.
I20260925 14:52:08.281950 11606 catalog_manager.cc:5495] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd62f7e4fbe4a4c8305988340469f93 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6dd62f7e4fbe4a4c8305988340469f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dd62f7e4fbe4a4c8305988340469f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:08.312168 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.308597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.98.3:40979)
0925 14:52:08.308654 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:08.308656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:08.308666 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:08.308790 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:08.308791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:08.308812 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:08.308901 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:08.309126 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.309881 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.311139 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:08.311438 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:08.311495 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:08.311700 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:08.311836 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:08.311905 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:08.312069 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:52:08.312086 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:52:08.313961 11770 raft_consensus.cc:1261] T 9fca7cbfb01c4e73b4108c969dd528ed P d510d202fe97483c8906e02b796b7e65 [term 1 FOLLOWER]: Refusing update from remote peer 6dd62f7e4fbe4a4c8305988340469f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:08.314301 12119 consensus_queue.cc:1035] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d510d202fe97483c8906e02b796b7e65" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:08.316531 11909 raft_consensus.cc:1261] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 1 FOLLOWER]: Refusing update from remote peer 6dd62f7e4fbe4a4c8305988340469f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:08.316963 12119 consensus_queue.cc:1035] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7a0cb13d334d2788c3d1de530bc553" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:08.317235 12122 mvcc.cc:205] Tried to move back new op lower bound from 7333265114371313664 to 7333265114239922176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265114371313664}]
I20260925 14:52:08.319041 12123 mvcc.cc:205] Tried to move back new op lower bound from 7333265114371313664 to 7333265114239922176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265114371313664}]
I20260925 14:52:08.320430 11606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:08.320528 11606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:08.321205 11606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41132e9224504096a6eeb7d2f2827c39: Sending DeleteTablet for 3 replicas of tablet 9fca7cbfb01c4e73b4108c969dd528ed
I20260925 14:52:08.321585 12024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33974
I20260925 14:52:08.321727 11750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34248
I20260925 14:52:08.321812 11876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fca7cbfb01c4e73b4108c969dd528ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46050
I20260925 14:52:08.322214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 11687
I20260925 14:52:08.330821 12127 mvcc.cc:205] Tried to move back new op lower bound from 7333265114371313664 to 7333265114239922176. Current Snapshot: MvccSnapshot[applied={T|T < 7333265114371313664}]
I20260925 14:52:08.331372 12131 tablet_replica.cc:330] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: stopping tablet replica
I20260925 14:52:08.331544 12133 tablet_replica.cc:330] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553: stopping tablet replica
I20260925 14:52:08.331689 12133 raft_consensus.cc:2229] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:08.331827 12131 raft_consensus.cc:2229] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:08.332012 12131 raft_consensus.cc:2258] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:08.332139 12133 raft_consensus.cc:2258] T 9fca7cbfb01c4e73b4108c969dd528ed P 5c7a0cb13d334d2788c3d1de530bc553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:08.332746 12131 ts_tablet_manager.cc:1905] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:08.335088 12131 ts_tablet_manager.cc:1918] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:08.335230 12131 log.cc:1196] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/9fca7cbfb01c4e73b4108c969dd528ed
I20260925 14:52:08.335565 12131 ts_tablet_manager.cc:1939] T 9fca7cbfb01c4e73b4108c969dd528ed P 6dd62f7e4fbe4a4c8305988340469f93: Deleting consensus metadata
W20260925 14:52:08.338522 11591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34811 (error 108)
I20260925 14:52:08.339293 11593 catalog_manager.cc:4841] TS 6dd62f7e4fbe4a4c8305988340469f93 (127.0.98.3:40979): tablet 9fca7cbfb01c4e73b4108c969dd528ed (table test-table [id=c9e1bdeebf684fdf8ef55c18d79d8049]) successfully deleted
I20260925 14:52:08.342479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 11824
I20260925 14:52:08.343725 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:08.343598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34811 (local address 127.0.0.1:34260)
0925 14:52:08.343653 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.343688 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:52:08.343842 11591 catalog_manager.cc:4584] TS d510d202fe97483c8906e02b796b7e65 (127.0.98.1:34811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fca7cbfb01c4e73b4108c969dd528ed: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34811: connect: Connection refused (error 111)
W20260925 14:52:08.350275 12104 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:08.349940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33015 (local address 127.0.0.1:46052)
0925 14:52:08.350042 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:08.350053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:08.350102 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:08.350220 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 14:52:08.350831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 11961
I20260925 14:52:08.357686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 11574
I20260925 14:52:08.363001 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:52:08.707829 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:52:08.707970 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:52:09.052767 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:52:09.052 UTC [11160] LOG: received smart shutdown request
2026-09-25 14:52:09.056 UTC [11160] LOG: background worker "logical replication launcher" (PID 11169) exited with exit code 1
2026-09-25 14:52:09.056 UTC [11164] LOG: shutting down
2026-09-25 14:52:09.214 UTC [11160] LOG: database system is shut down
2026-09-25T14:52:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45829 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:09 dist-test-slave-gpc1 krb5kdc[12144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:09 dist-test-slave-gpc1 krb5kdc[12144](info): set up 2 sockets
Sep 25 14:52:09 dist-test-slave-gpc1 krb5kdc[12144](info): commencing operation
krb5kdc: starting...
W20260925 14:52:10.410480 392 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:52:10 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 14:52:10.572429 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbEnjDj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/postgres -l logfile start
2026-09-25 14:52:13.009 UTC [12174] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:52:13.009 UTC [12174] LOG: listening on IPv4 address "127.0.98.20", port 60267
2026-09-25 14:52:13.012 UTC [12174] LOG: listening on Unix socket "/tmp/.s.PGSQL.60267"
2026-09-25 14:52:13.032 UTC [12177] LOG: database system was shut down at 2026-09-25 14:52:11 UTC
2026-09-25 14:52:13.036 UTC [12174] LOG: database system is ready to accept connections
I20260925 14:52:14.022457 392 mini_postgres.cc:96] Postgres bound to 60267
2026-09-25 14:52:14.028 UTC [12185] FATAL: database "slave" does not exist
127.0.98.20:60267 - accepting connections
I20260925 14:52:14.029070 392 mini_ranger.cc:161] Starting Ranger...
I20260925 14:52:14.043808 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:52:14.493129 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:52:14.493196 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin
2026-09-25 14:52:14,777 [I] DB FLAVOR :POSTGRES
2026-09-25 14:52:14,777 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:52:14,777 [I] Checking connection..
2026-09-25 14:52:14,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:52:15,033 [I] Checking connection passed.
2026-09-25 14:52:15,033 [I] --------- Verifying version history table ---------
2026-09-25 14:52:15,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:52:15,289 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:52:15,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:52:15,558 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:52:15,559 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:52:15,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:52:15.821 UTC [12249] WARNING: there is no transaction in progress
2026-09-25 14:52:15,827 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:52:15,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:52:16,083 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:52:16,084 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:52:16,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:52:16,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:52:16,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gpc1.c.gcp-upstream.internal','N') ;"
2026-09-25 14:52:16,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:52:17,041 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:52:17,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:52:17,285 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:52:17,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:52:17,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:52:17,746 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:52:17,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:52:18.744 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.762 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.779 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.796 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.813 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.834 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.894 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.901 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.917 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.928 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.940 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.956 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.966 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.973 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.980 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:18.987 UTC [12357] WARNING: there is no transaction in progress
2026-09-25 14:52:19,630 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:52:19,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:52:19,865 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:52:19,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:52:20,127 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:52:20,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:52:20,394 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:52:20,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:52:20,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:52:20,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gpc1.c.gcp-upstream.internal';"
2026-09-25 14:52:21,148 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:52:21,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:52:21,381 [I] DB_PATCHES have already been applied
I20260925 14:52:21.387104 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbEnjDj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:52:21.387171 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 14:52:21.390132 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:35385
Sep 25, 2026 2:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:52:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:52:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:52:22.563446 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.534s user 0.000s sys 0.008s
Sep 25, 2026 2:52:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:52:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 2:52:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:52:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:52:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:52:44.775977 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:52:44 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347964, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:52:44.858656 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:52:44.867594 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:52:45.735257 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:52:45.735342 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms
I20260925 14:52:45.735364 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60267
2026-09-25 14:52:45,898 [I] DB FLAVOR :POSTGRES
2026-09-25 14:52:45,899 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:52:45,899 [I] Checking connection
2026-09-25 14:52:45,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:52:46,140 [I] connection success
2026-09-25 14:52:46,141 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:52:46,141 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:52:46,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:52:46,372 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:52:46,372 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:52:46,373 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:52:46,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:52:46,651 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:52:47.168529 392 mini_ranger_kms.cc:310] Created kms service
I20260925 14:52:47.387327 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:52:47.434808 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:52:47 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347964, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:47.837934 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:52:47.838058 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbEnjDj/build/release/bin/postgresql.jar:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbEnjDj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:52:47.838075 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 14:52:47.842100 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12537
I20260925 14:52:47.842197 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55845
Sep 25, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:52:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:52:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:52:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:52:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:52:50.144945 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.286s user 0.000s sys 0.146s
I20260925 14:52:50.145048 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:52:50.145107 392 mini_ranger_kms.cc:401] 127.0.98.20:55845/kms/v1/keys
Sep 25 14:52:50 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:52:50 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:52:51 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347970, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:52:53 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:53.641505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43165
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:53.722337 12588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.722604 12588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.722625 12588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.723764 12588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:53.723815 12588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.723830 12588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.723842 12588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:53.723855 12588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:53.725423 12588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43165
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:53.725662 12588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.725915 12588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.732141 12594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.732184 12596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.732324 12593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.732220 12588 server_base.cc:1034] running on GCE node
I20260925 14:52:53.732657 12588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.732975 12588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.734149 12588 hybrid_clock.cc:648] HybridClock initialized: now 1790347973734133 us; error 33 us; skew 500 ppm
Sep 25 14:52:53 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.749046 12588 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:53.749663 12588 webserver.cc:460] Webserver started at http://127.0.98.62:42667/ using document root <none> and password file <none>
I20260925 14:52:53.749920 12588 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.749984 12588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.750144 12588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:53 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:53.795025 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "0d6a6f6194384ea0ac85f1d792b16bb1"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-gpc1"
server_key: "1affd1683944086c30f22b672fee5cd5"
server_key_iv: "88af24b21d8f4e61448cfd1620c48f00"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:53.795439 12588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d6a6f6194384ea0ac85f1d792b16bb1"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-gpc1"
server_key: "1affd1683944086c30f22b672fee5cd5"
server_key_iv: "88af24b21d8f4e61448cfd1620c48f00"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:53.796963 12588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:53.808878 12603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.809074 12588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.809203 12588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0d6a6f6194384ea0ac85f1d792b16bb1"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-gpc1"
server_key: "1affd1683944086c30f22b672fee5cd5"
server_key_iv: "88af24b21d8f4e61448cfd1620c48f00"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:53.809295 12588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.832522 12588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.834664 12588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.834822 12588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.839910 12588 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43165
I20260925 14:52:53.839942 12655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43165 every 8 connection(s)
I20260925 14:52:53.840421 12588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:53.841101 12656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:53.843461 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Bootstrap starting.
I20260925 14:52:53.844120 12656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:53.844419 12656 log.cc:824] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:53.845132 12656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: No bootstrap required, opened a new log
I20260925 14:52:53.847249 12656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:53.847427 12656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:53.847476 12656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6a6f6194384ea0ac85f1d792b16bb1, State: Initialized, Role: FOLLOWER
I20260925 14:52:53.847708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 12588
I20260925 14:52:53.847697 12656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:53.847781 12656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:53.847822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:53.847862 12656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:53.847909 12656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.848037 392 external_mini_cluster.cc:1428] Setting key 30d5fb42136e22461ad8014d05c476ff
I20260925 14:52:53.848649 12656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:53.848827 12656 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d6a6f6194384ea0ac85f1d792b16bb1; no voters:
I20260925 14:52:53.849020 12656 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:53.849079 12661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:53.849287 12661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 0d6a6f6194384ea0ac85f1d792b16bb1, State: Running, Role: LEADER
I20260925 14:52:53.849372 12656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:53.849452 12661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:53.849836 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260925 14:52:53.849973 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.850157 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d6a6f6194384ea0ac85f1d792b16bb1. Latest consensus state: current_term: 1 leader_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260925 14:52:53.850286 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.850483 12670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:53.850991 12670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:53.851302 12656 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:52:53.852710 12670 catalog_manager.cc:1347] Generated new cluster ID: 81bc2135da284736922b29cd16f2aba4
I20260925 14:52:53.852753 12670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:52:53 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:53.864140 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.848543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.0.98.62:43165)
0925 14:52:53.848719 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.848729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.848763 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:52:53.848896 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.848904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.848964 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.849969 (+ 1005us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.850085 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.851644 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.852711 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.853459 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.856463 (+ 3004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.856554 (+ 91us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.856558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.856558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.856567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.858610 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.859069 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.859073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.859074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.859208 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.859460 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.859460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.859461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.859766 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.862812 (+ 3046us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.863045 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.863752 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260925 14:52:53.869863 12670 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:53.871043 12670 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:53.875609 12670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Generated new TSK 0
I20260925 14:52:53.875856 12670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:52:54.588644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:54.676013 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:54.676245 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:54.676291 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:54.677454 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:54.677520 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:54.677564 12701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:54.679147 12701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:54.679416 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:54.679643 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:54.685534 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.685542 12709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.685847 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:54.686007 12701 server_base.cc:1034] running on GCE node
I20260925 14:52:54.686213 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:54.686504 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:54.687677 12701 hybrid_clock.cc:648] HybridClock initialized: now 1790347974687655 us; error 32 us; skew 500 ppm
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:54.690828 12701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:54.691370 12701 webserver.cc:460] Webserver started at http://127.0.98.1:35429/ using document root <none> and password file <none>
I20260925 14:52:54.691606 12701 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:54.691672 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:54.691821 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:54.706534 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "c719d6c0c08944b59997a66b01c38e9e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "fcac4be9836af18a9cba13ae29e0d368"
server_key_iv: "234618ce77df4152626ce5a8ae0b0166"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.707098 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c719d6c0c08944b59997a66b01c38e9e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "fcac4be9836af18a9cba13ae29e0d368"
server_key_iv: "234618ce77df4152626ce5a8ae0b0166"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.710803 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 14:52:54.728333 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.728645 12701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:52:54.728818 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "c719d6c0c08944b59997a66b01c38e9e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "fcac4be9836af18a9cba13ae29e0d368"
server_key_iv: "234618ce77df4152626ce5a8ae0b0166"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.728999 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:54.751811 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:54.754488 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:54.754717 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:54.755033 12701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:54.755463 12701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:54.755518 12701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.755553 12701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:54.755573 12701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.761845 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37773
I20260925 14:52:54.764150 12831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37773 every 8 connection(s)
I20260925 14:52:54.762315 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:54.768071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 12701
I20260925 14:52:54.768172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:54.768368 392 external_mini_cluster.cc:1428] Setting key d68661c3a940dba0b690398403caf942
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:54.779068 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.767667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.1:39587)
0925 14:52:54.768044 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.768092 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.768735 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.769153 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.769158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.769408 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.769690 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.769696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.770762 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.770763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.772065 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.772067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.772609 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.772610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.772751 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.773423 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:54.773434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.775220 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.777326 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.777333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.777335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.777622 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.777857 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:54.777858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:54.777858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:54.777986 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:54.778323 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.778330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.778535 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.778666 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.778827 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":82,"threads_started":1}
I20260925 14:52:54.779348 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.767975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39587 (local address 127.0.98.62:43165)
0925 14:52:54.768924 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.768926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.768933 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:52:54.768955 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.768956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.768968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.769788 (+ 820us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.769828 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.770650 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.772177 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.772497 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.774095 (+ 1598us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.775329 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.775331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.775331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:54.775339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.777228 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.777724 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.777725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.777725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.777787 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:54.778059 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:54.778060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:54.778060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:54.778193 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.779112 (+ 919us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.779158 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.779244 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":80,"threads_started":1}
I20260925 14:52:54.779933 12832 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:54.780031 12832 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.780217 12832 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:54.780777 12620 ts_manager.cc:194] Registered new tserver with Master: c719d6c0c08944b59997a66b01c38e9e (127.0.98.1:37773)
I20260925 14:52:54.781678 12620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39587
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:52:54.806995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:52:54.922832 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:54.923103 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:54.923139 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:54.924927 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:54.925009 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:54.925068 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:54.927313 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:54.927718 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:54.928053 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:54.935163 12848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.935413 12846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.935532 12845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:54.936794 12838 server_base.cc:1034] running on GCE node
I20260925 14:52:54.937027 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:54.937290 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:54.938427 12838 hybrid_clock.cc:648] HybridClock initialized: now 1790347974938389 us; error 50 us; skew 500 ppm
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:54.941802 12838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:54.942308 12838 webserver.cc:460] Webserver started at http://127.0.98.2:34343/ using document root <none> and password file <none>
I20260925 14:52:54.942572 12838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:54.942622 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:54.942756 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:54 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:54.961843 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "41a81e57952f416cbaa86fe5b70c166e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "5ec56eede0674b92a5bb8e7d0bc783db"
server_key_iv: "34819b0330c15a3aba22fba7c722b6c1"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.962600 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "41a81e57952f416cbaa86fe5b70c166e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "5ec56eede0674b92a5bb8e7d0bc783db"
server_key_iv: "34819b0330c15a3aba22fba7c722b6c1"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.965965 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 14:52:54.987389 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.987668 12838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:54.987761 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "41a81e57952f416cbaa86fe5b70c166e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "5ec56eede0674b92a5bb8e7d0bc783db"
server_key_iv: "34819b0330c15a3aba22fba7c722b6c1"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:54.987874 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.014117 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.016760 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.016975 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.017258 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.017711 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.017802 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.017875 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.017933 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.024526 12838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40467
I20260925 14:52:55.025012 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:55.026837 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40467 every 8 connection(s)
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.034848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 12838
I20260925 14:52:55.034953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:55.035171 392 external_mini_cluster.cc:1428] Setting key 74ef44c7ca4d61b88f91a45721eda9f1
I20260925 14:52:55.037415 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.028068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54863 (local address 127.0.98.62:43165)
0925 14:52:55.028115 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.028117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.028316 (+ 199us) server_negotiation.cc:407] Connection header received
0925 14:52:55.029008 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.029010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.029022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.029054 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.029902 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.030602 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.031603 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.031812 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.031903 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.033971 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.033972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.033972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.033979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.035466 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.036244 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.036245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.036246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.036303 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.036586 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.036587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.036587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.036740 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.036783 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.037168 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.037268 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:52:55.038233 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.027775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.2:54863)
0925 14:52:55.028191 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.028235 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.028881 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.029249 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.029253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.029500 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.029782 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.029788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.030696 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.030697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.031487 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.031489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.031924 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.031925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.032053 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.032597 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.032606 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:55.033868 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:55.035830 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.035837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.035839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.036158 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.036373 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.036374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.036374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.036517 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.036829 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:55.036834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:55.037019 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.037787 (+ 768us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.037968 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":99,"threads_started":1}
I20260925 14:52:55.039165 12970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:55.039304 12970 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.039537 12970 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:55.039923 12620 ts_manager.cc:194] Registered new tserver with Master: 41a81e57952f416cbaa86fe5b70c166e (127.0.98.2:40467)
I20260925 14:52:55.040452 12620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54863
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:52:55.065464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:55.181066 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.181418 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.181504 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.183305 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.183418 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.183501 12975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:55.185830 12975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:55.186373 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.186746 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.195887 12981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.196727 12980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.198971 12983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.199685 12975 server_base.cc:1034] running on GCE node
I20260925 14:52:55.199944 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.200258 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.201428 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790347975201407 us; error 41 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.204831 12975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:55.205368 12975 webserver.cc:460] Webserver started at http://127.0.98.3:37637/ using document root <none> and password file <none>
I20260925 14:52:55.205673 12975 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.205732 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:55.205891 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:55.219703 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e21dfdbc5464691a7197aa2ae939fc6"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-gpc1"
server_key: "c5c7b16bbc774cf04397976ff1469974"
server_key_iv: "acb26b42b50ca8de61d5e26e874b9efc"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.220182 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e21dfdbc5464691a7197aa2ae939fc6"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-gpc1"
server_key: "c5c7b16bbc774cf04397976ff1469974"
server_key_iv: "acb26b42b50ca8de61d5e26e874b9efc"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.222225 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:52:55.244833 12990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.245095 12975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:52:55.245213 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "7e21dfdbc5464691a7197aa2ae939fc6"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-gpc1"
server_key: "c5c7b16bbc774cf04397976ff1469974"
server_key_iv: "acb26b42b50ca8de61d5e26e874b9efc"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.245335 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.258940 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.261556 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.261754 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.262025 12975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.262431 12975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.262552 12975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.262627 12975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.262683 12975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.271925 12975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34145
I20260925 14:52:55.272442 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:55.274945 13103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34145 every 8 connection(s)
I20260925 14:52:55.276423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 12975
I20260925 14:52:55.276525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:55.276729 392 external_mini_cluster.cc:1428] Setting key efed9b41965d66da69bdbd45db6cb35e
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.286458 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.275898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55849 (local address 127.0.98.62:43165)
0925 14:52:55.275962 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.275964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.276226 (+ 262us) server_negotiation.cc:407] Connection header received
0925 14:52:55.277246 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.277247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.277260 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.277289 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.278116 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.278924 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.280388 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.280678 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.280717 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.283319 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.283321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.283321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.283328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.284944 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.285414 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.285415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.285416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.285475 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.285731 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.285732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.285732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.285859 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.285898 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.286271 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.286368 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:52:55.287205 13106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.275622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.3:55849)
0925 14:52:55.276089 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.276133 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.277119 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.277476 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.277481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.277732 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.278012 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.278018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.279036 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.279036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.280288 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.280291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.280817 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.280817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.280954 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.281579 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.281589 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:55.283213 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:55.285037 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.285043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.285045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.285331 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.285541 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.285542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.285542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.285663 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.285938 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:55.285943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:55.286113 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.286818 (+ 705us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.286972 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":82,"threads_started":1}
I20260925 14:52:55.288131 13104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:55.288242 13104 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.288455 13104 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:55.288928 12620 ts_manager.cc:194] Registered new tserver with Master: 7e21dfdbc5464691a7197aa2ae939fc6 (127.0.98.3:34145)
I20260925 14:52:55.289518 12620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55849
I20260925 14:52:55.291915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:55.292680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 12701
I20260925 14:52:55.303660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 12838
I20260925 14:52:55.309597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 12975
I20260925 14:52:55.315044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 12588
I20260925 14:52:55.322536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--webserver_interface=127.0.98.62
--webserver_port=42667
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43165
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:55.403021 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.403237 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.403259 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.404419 13107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:55.404461 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.404475 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.404489 13107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:55.404501 13107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:55.406644 13107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43165
--ranger_config_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:55.406908 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.407132 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.411995 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.411995 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.412252 13107 server_base.cc:1034] running on GCE node
W20260925 14:52:55.412012 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.412578 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.412886 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.414054 13107 hybrid_clock.cc:648] HybridClock initialized: now 1790347975414041 us; error 29 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.416692 13107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:55.417172 13107 webserver.cc:460] Webserver started at http://127.0.98.62:42667/ using document root <none> and password file <none>
I20260925 14:52:55.417392 13107 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.417466 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:55.472975 13107 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:55.473856 13123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.474051 13107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:55.474159 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0d6a6f6194384ea0ac85f1d792b16bb1"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-gpc1"
server_key: "1affd1683944086c30f22b672fee5cd5"
server_key_iv: "88af24b21d8f4e61448cfd1620c48f00"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.474675 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.490898 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.492637 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.492750 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.497037 13107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43165
I20260925 14:52:55.497080 13175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43165 every 8 connection(s)
I20260925 14:52:55.497454 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:55.498759 13176 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:52:55.498850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13107
I20260925 14:52:55.499130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37773
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43165
--webserver_port=35429
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:55.499780 13176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Bootstrap starting.
I20260925 14:52:55.507943 13176 log.cc:824] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:55.509716 13176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:55.510015 13176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Bootstrap complete.
I20260925 14:52:55.512501 13176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:55.512739 13176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6a6f6194384ea0ac85f1d792b16bb1, State: Initialized, Role: FOLLOWER
I20260925 14:52:55.512903 13176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:55.512981 13176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:55.513010 13176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:55.513075 13176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:52:55.514053 13176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:55.514168 13176 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d6a6f6194384ea0ac85f1d792b16bb1; no voters:
I20260925 14:52:55.514375 13176 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:52:55.514410 13180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:52:55.514552 13180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [term 2 LEADER]: Becoming Leader. State: Replica: 0d6a6f6194384ea0ac85f1d792b16bb1, State: Running, Role: LEADER
I20260925 14:52:55.514657 13176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:55.514706 13180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } }
I20260925 14:52:55.515004 13180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d6a6f6194384ea0ac85f1d792b16bb1. Latest consensus state: current_term: 2 leader_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260925 14:52:55.515008 13181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d6a6f6194384ea0ac85f1d792b16bb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43165 } } }
I20260925 14:52:55.515126 13180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:55.515233 13181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:55.515443 13189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:55.517313 13176 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:52:55.516971 13189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:55.517825 13189 catalog_manager.cc:1259] Loaded cluster ID: 81bc2135da284736922b29cd16f2aba4
I20260925 14:52:55.517889 13189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:55.519163 13189 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:55.519397 13189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Loaded TSK: 0
I20260925 14:52:55.519662 13189 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:52:55.608383 13178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.608667 13178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.608736 13178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.610302 13178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.610394 13178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.610440 13178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:55.611948 13178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35429
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:55.612216 13178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.612493 13178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.617930 13216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.618063 13213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.617946 13214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.618980 13178 server_base.cc:1034] running on GCE node
I20260925 14:52:55.619303 13178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.619580 13178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.626957 13178 hybrid_clock.cc:648] HybridClock initialized: now 1790347975626924 us; error 40 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.629802 13178 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:55.630344 13178 webserver.cc:460] Webserver started at http://127.0.98.1:35429/ using document root <none> and password file <none>
I20260925 14:52:55.630641 13178 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.630779 13178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:55.662808 13178 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.000s sys 0.001s
I20260925 14:52:55.663664 13224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.663851 13178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:55.663931 13178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "c719d6c0c08944b59997a66b01c38e9e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "fcac4be9836af18a9cba13ae29e0d368"
server_key_iv: "234618ce77df4152626ce5a8ae0b0166"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.664434 13178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.682238 13178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.684916 13178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.685141 13178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.685381 13178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.685804 13178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.685861 13178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.685894 13178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.685948 13178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.693104 13178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37773
I20260925 14:52:55.693222 13337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37773 every 8 connection(s)
I20260925 14:52:55.693616 13178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:55.696322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13178
I20260925 14:52:55.696606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40467
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43165
--webserver_port=34343
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.707924 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.694423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.1:42757)
0925 14:52:55.694901 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.694948 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.695724 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.696424 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.696427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.696777 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.697051 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.697057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.698849 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.698850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.700118 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.700121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.701087 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.701088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.701223 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.701834 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.701844 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:55.703361 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:55.705779 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.705785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.705788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.706116 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.706467 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.706468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.706468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.706609 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.707145 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:55.707151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:55.707332 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.707447 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.707619 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":77,"threads_started":1}
I20260925 14:52:55.708622 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.694527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42757 (local address 127.0.98.62:43165)
0925 14:52:55.694750 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.694760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.695085 (+ 325us) server_negotiation.cc:407] Connection header received
0925 14:52:55.695984 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.695992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.696060 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.696253 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.697242 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.698730 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.700230 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.700874 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.700981 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.703481 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.703485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.703486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.703498 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.705680 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.706228 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.706232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.706232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.706385 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.706691 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.706691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.706692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.707017 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.707101 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.708245 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.708378 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260925 14:52:55.711174 13338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:55.711298 13338 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.711488 13338 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:55.712472 13338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43165 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:52:55.712548 13338 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.712608 13338 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:55.714753 13338 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.714833 13338 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:55.715247 13338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:52:55.819890 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.820231 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.820293 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.822049 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.822137 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.822196 13342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:55.824456 13342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34343
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:55.824743 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.825069 13342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.831035 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.831050 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.831537 13350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.832261 13342 server_base.cc:1034] running on GCE node
I20260925 14:52:55.832479 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.832751 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.836027 13342 hybrid_clock.cc:648] HybridClock initialized: now 1790347975835294 us; error 747 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.839123 13342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:55.839753 13342 webserver.cc:460] Webserver started at http://127.0.98.2:34343/ using document root <none> and password file <none>
I20260925 14:52:55.839993 13342 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.840075 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:55.859426 13342 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:55.860262 13358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.860481 13342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:55.860575 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "41a81e57952f416cbaa86fe5b70c166e"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-gpc1"
server_key: "5ec56eede0674b92a5bb8e7d0bc783db"
server_key_iv: "34819b0330c15a3aba22fba7c722b6c1"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:55.860970 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.883646 13342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.886143 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.886310 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.886623 13342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.886963 13342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.887018 13342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.887063 13342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.887082 13342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.894410 13342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40467
I20260925 14:52:55.894836 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:55.895462 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40467 every 8 connection(s)
I20260925 14:52:55.896349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13342
I20260925 14:52:55.896637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43165
--webserver_port=37637
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:52:55 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.906458 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.895636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.2:47287)
0925 14:52:55.895805 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.895849 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.896535 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.896947 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.896952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.897208 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.897497 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.897503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.898581 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.898583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.899642 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.899644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.900211 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.900212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.900358 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.900949 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.900961 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:55.902567 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:55.904717 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.904724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.904726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.905061 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.905296 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.905297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.905297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.905420 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.905728 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:55.905735 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:55.905915 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.906031 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.906190 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260925 14:52:55.906816 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.895966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47287 (local address 127.0.98.62:43165)
0925 14:52:55.896005 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.896007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.896017 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:55.896698 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.896700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.896717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.896752 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.897623 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.898444 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.899742 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.900061 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.900130 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.902701 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.902702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.902703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.902711 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.904588 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.905161 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.905162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.905163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.905226 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.905487 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.905488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.905489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.905619 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.905697 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.906626 (+ 929us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.906727 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:52:55.907215 13472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:55.907330 13472 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.907545 13472 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:55.908291 13472 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43165 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:52:55.908427 13472 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.908519 13472 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:55.908906 13472 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.909029 13472 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:55.909407 13472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:52:55.999094 13475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.999432 13475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.999500 13475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:56.000773 13475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:56.000836 13475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:56.000882 13475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:56.002887 13475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37637
--tserver_master_addrs=127.0.98.62:43165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:56.003275 13475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:56.003530 13475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:56.008539 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.008836 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.009035 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:56.009251 13475 server_base.cc:1034] running on GCE node
I20260925 14:52:56.009454 13475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:56.009732 13475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:56.010888 13475 hybrid_clock.cc:648] HybridClock initialized: now 1790347976010860 us; error 46 us; skew 500 ppm
Sep 25 14:52:56 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:56.013698 13475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:56.014312 13475 webserver.cc:460] Webserver started at http://127.0.98.3:37637/ using document root <none> and password file <none>
I20260925 14:52:56.014581 13475 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:56.014722 13475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:52:56 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:56.032037 13475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:56.032799 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.033003 13475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:56.033094 13475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "7e21dfdbc5464691a7197aa2ae939fc6"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-gpc1"
server_key: "c5c7b16bbc774cf04397976ff1469974"
server_key_iv: "acb26b42b50ca8de61d5e26e874b9efc"
server_key_version: "kuduclusterkey@0"
I20260925 14:52:56.033473 13475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:56.052500 13475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:56.054826 13475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:56.054997 13475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:56.055274 13475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:56.055644 13475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:56.055696 13475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.055765 13475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:56.055794 13475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.063817 13475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34145
I20260925 14:52:56.063927 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34145 every 8 connection(s)
I20260925 14:52:56.064323 13475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:56.064893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13475
Sep 25 14:52:56 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:56.078159 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.065627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57471 (local address 127.0.98.62:43165)
0925 14:52:56.065754 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.065756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.065769 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:52:56.066837 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.066840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.066853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.066932 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.067892 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.068659 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.070207 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.070512 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.070594 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.074540 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.074541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.074541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:56.074549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.076343 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.076902 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.076903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.076904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.076962 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.077313 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.077315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.077315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.077478 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.077527 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.077967 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.078057 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 14:52:56.078415 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.065293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43165 (local address 127.0.98.3:57471)
0925 14:52:56.065524 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.065579 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.066696 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.067093 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.067097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.067381 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:56.067707 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.067727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.068774 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.068775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.070054 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.070057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.070672 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.070673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.070820 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.071373 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:56.071384 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:56.074420 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:56.076444 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.076451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.076453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.076790 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.077048 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.077049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.077049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.077198 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.077578 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:56.077585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:56.077794 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.077942 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.078104 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260925 14:52:56.079061 13606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43165
I20260925 14:52:56.079169 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.079397 13606 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:56.080160 13606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43165 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:52:56.080246 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.080318 13606 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:56.080720 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.080794 13606 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
W20260925 14:52:56.081156 13606 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:52:56.081844 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.065648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.0.98.62:43165)
0925 14:52:56.066920 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.066922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.066930 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:56.066956 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.066957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.066969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.067930 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.067971 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.068632 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.069701 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.069936 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.070025 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.078399 (+ 8374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.078400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.078401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:56.078408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.080556 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.080997 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.080999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.080999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.081060 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.081323 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.081324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.081325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.081446 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.081576 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.081697 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.081771 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":70,"threads_started":1}
Sep 25 14:52:56 dist-test-slave-gpc1 krb5kdc[12144](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:56.715437 13338 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.715608 13338 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:56.715986 13140 ts_manager.cc:194] Registered new tserver with Master: c719d6c0c08944b59997a66b01c38e9e (127.0.98.1:37773)
I20260925 14:52:56.717032 13140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42757
I20260925 14:52:56.909616 13472 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.909780 13472 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:56.910154 13140 ts_manager.cc:194] Registered new tserver with Master: 41a81e57952f416cbaa86fe5b70c166e (127.0.98.2:40467)
I20260925 14:52:56.910787 13140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47287
Sep 25 14:52:56 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:52:57.081373 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:57.081532 13606 heartbeater.cc:511] Master 127.0.98.62:43165 requested a full tablet report, sending...
I20260925 14:52:57.081841 13140 ts_manager.cc:194] Registered new tserver with Master: 7e21dfdbc5464691a7197aa2ae939fc6 (127.0.98.3:34145)
I20260925 14:52:57.082413 13140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57471
I20260925 14:52:57.088842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:57.101156 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.089813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.0.98.62:43165)
0925 14:52:57.089969 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.089971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.094911 (+ 4940us) server_negotiation.cc:407] Connection header received
0925 14:52:57.094938 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.094939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.094952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.095007 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.095676 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.096179 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.096895 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.097102 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.097176 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.098472 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.098473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.098473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:57.098481 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.100006 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.100366 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.100368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.100368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.100425 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:57.100625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:57.100626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:57.100626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:57.100739 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.100845 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.100951 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.101033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260925 14:52:57.103205 13140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:57.104884 13140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:57.118181 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.98.2:40467)
0925 14:52:57.114442 (+ 4747us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.114448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.114660 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:52:57.114701 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.114704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.114721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.114799 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.115221 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.116034 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.117159 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.117648 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.117746 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.117838 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.118067 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.118115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4704,"thread_start_us":55,"threads_started":1}
I20260925 14:52:57.118482 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40467 (local address 127.0.0.1:34052)
0925 14:52:57.114440 (+ 5061us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.114494 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.114576 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.114837 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.114840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.114842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.114962 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.114964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.116133 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.116134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.117055 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.117057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.117748 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.117750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.117937 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.117953 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.118323 (+ 370us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.118411 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5019,"thread_start_us":74,"threads_started":1}
I20260925 14:52:57.119165 13406 tablet_service.cc:1467] Processing CreateTablet for tablet 9406831ef82e4db1a900055491a35e3f (DEFAULT_TABLE table=test-table [id=8e1cd6c025f7480c9cb56925d642b9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.119508 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9406831ef82e4db1a900055491a35e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.122264 13640 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Bootstrap starting.
I20260925 14:52:57.122934 13640 tablet_bootstrap.cc:654] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.123234 13640 log.cc:824] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.123921 13640 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: No bootstrap required, opened a new log
I20260925 14:52:57.124002 13640 ts_tablet_manager.cc:1397] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:57.126219 13640 raft_consensus.cc:348] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.126410 13640 raft_consensus.cc:374] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.126472 13640 raft_consensus.cc:729] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a81e57952f416cbaa86fe5b70c166e, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.126629 13640 consensus_queue.cc:260] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.126919 13640 ts_tablet_manager.cc:1428] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:57.127048 13472 heartbeater.cc:503] Master 127.0.98.62:43165 was elected leader, sending a full tablet report...
I20260925 14:52:57.130911 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.98.3:34145)
0925 14:52:57.126681 (+ 16990us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.126685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.126694 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:52:57.126724 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.126726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.126740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.126963 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.127039 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.128131 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.129503 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.130440 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.130491 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.130560 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.130777 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.130825 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16954,"thread_start_us":52,"threads_started":1}
I20260925 14:52:57.131306 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34145 (local address 127.0.0.1:41814)
0925 14:52:57.116395 (+ 6958us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.116407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.116462 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.126840 (+ 10378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.126841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.126844 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.126897 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.126898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.128213 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.128214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.129391 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.129394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.130619 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.130620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.130682 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.130686 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.131168 (+ 482us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.131216 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6940,"thread_start_us":88,"threads_started":1}
I20260925 14:52:57.132089 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37773 (local address 127.0.0.1:50332)
0925 14:52:57.114536 (+ 5147us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.114565 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.114625 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.128276 (+ 13651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.128277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.128279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.128319 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.128320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.129540 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.129541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.130647 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.130649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.131853 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.131854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.131901 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.131904 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.131976 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.132018 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5113,"thread_start_us":68,"threads_started":1}
I20260925 14:52:57.132102 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.109643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.0.98.1:37773)
0925 14:52:57.126440 (+ 16797us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.126446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.126460 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:57.126495 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.126497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.126515 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.128271 (+ 1756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.128416 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.129454 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.130887 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.131663 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.131733 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.131822 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.131991 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.132030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16753,"thread_start_us":61,"threads_started":1}
I20260925 14:52:57.132242 13540 tablet_service.cc:1467] Processing CreateTablet for tablet 9406831ef82e4db1a900055491a35e3f (DEFAULT_TABLE table=test-table [id=8e1cd6c025f7480c9cb56925d642b9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.132611 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9406831ef82e4db1a900055491a35e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.132798 13272 tablet_service.cc:1467] Processing CreateTablet for tablet 9406831ef82e4db1a900055491a35e3f (DEFAULT_TABLE table=test-table [id=8e1cd6c025f7480c9cb56925d642b9da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.133086 13272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9406831ef82e4db1a900055491a35e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.136204 13643 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Bootstrap starting.
I20260925 14:52:57.136859 13643 tablet_bootstrap.cc:654] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.137853 13644 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Bootstrap starting.
I20260925 14:52:57.138464 13644 tablet_bootstrap.cc:654] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.138657 13643 log.cc:824] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.138870 13644 log.cc:824] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.139743 13644 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: No bootstrap required, opened a new log
I20260925 14:52:57.139813 13644 ts_tablet_manager.cc:1397] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:57.142271 13644 raft_consensus.cc:348] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.142810 13643 tablet_bootstrap.cc:492] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: No bootstrap required, opened a new log
I20260925 14:52:57.142865 13643 ts_tablet_manager.cc:1397] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 14:52:57.145332 13643 raft_consensus.cc:348] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.145484 13643 raft_consensus.cc:374] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.145512 13643 raft_consensus.cc:729] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e21dfdbc5464691a7197aa2ae939fc6, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.145779 13644 raft_consensus.cc:374] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.145892 13644 raft_consensus.cc:729] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c719d6c0c08944b59997a66b01c38e9e, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.145864 13643 consensus_queue.cc:260] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.146096 13644 consensus_queue.cc:260] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
W20260925 14:52:57.146222 13473 tablet.cc:2369] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:57.146474 13644 ts_tablet_manager.cc:1428] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260925 14:52:57.146585 13643 ts_tablet_manager.cc:1428] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:52:57.146639 13338 heartbeater.cc:503] Master 127.0.98.62:43165 was elected leader, sending a full tablet report...
I20260925 14:52:57.146785 13606 heartbeater.cc:503] Master 127.0.98.62:43165 was elected leader, sending a full tablet report...
W20260925 14:52:57.195163 13339 tablet.cc:2369] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:52:57 dist-test-slave-gpc1 krb5kdc[12144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260925 14:52:57.315814 13607 tablet.cc:2369] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:57.375568 13642 raft_consensus.cc:482] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:57.375722 13642 raft_consensus.cc:504] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.376111 13642 leader_election.cc:288] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c719d6c0c08944b59997a66b01c38e9e (127.0.98.1:37773), 7e21dfdbc5464691a7197aa2ae939fc6 (127.0.98.3:34145)
I20260925 14:52:57.388125 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.376718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54809 (local address 127.0.98.1:37773)
0925 14:52:57.376782 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.376785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.378555 (+ 1770us) server_negotiation.cc:407] Connection header received
0925 14:52:57.378666 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.378667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.378695 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.378725 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.378977 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.379764 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.387025 (+ 7261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.387676 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.387732 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.387744 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.387992 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.388043 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:52:57.388679 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.376395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37773 (local address 127.0.98.2:54809)
0925 14:52:57.378466 (+ 2071us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.378479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.378588 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.378807 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.378808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.378810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.378905 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.378907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.380501 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.380502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.381791 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.381794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.387833 (+ 6039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.387834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.387910 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.387914 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.388515 (+ 601us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.388580 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050,"thread_start_us":104,"threads_started":1}
I20260925 14:52:57.388692 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.377026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49579 (local address 127.0.98.3:34145)
0925 14:52:57.377080 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.377083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.385039 (+ 7956us) server_negotiation.cc:407] Connection header received
0925 14:52:57.385154 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.385156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.385169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.385201 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.385447 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.386189 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.387637 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.388267 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.388313 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.388321 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.388573 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.388619 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:52:57.388947 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.376854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34145 (local address 127.0.98.2:49579)
0925 14:52:57.384952 (+ 8098us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.384967 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.385072 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.385275 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.385276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.385278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.385353 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.385354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.386294 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.386295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.387530 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.387533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.388420 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.388421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.388495 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.388497 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.388827 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.388870 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8074,"thread_start_us":80,"threads_started":1}
I20260925 14:52:57.389271 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9406831ef82e4db1a900055491a35e3f" candidate_uuid: "41a81e57952f416cbaa86fe5b70c166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c719d6c0c08944b59997a66b01c38e9e" is_pre_election: true
I20260925 14:52:57.389392 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9406831ef82e4db1a900055491a35e3f" candidate_uuid: "41a81e57952f416cbaa86fe5b70c166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" is_pre_election: true
I20260925 14:52:57.389454 13292 raft_consensus.cc:2454] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a81e57952f416cbaa86fe5b70c166e in term 0.
I20260925 14:52:57.389537 13560 raft_consensus.cc:2454] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a81e57952f416cbaa86fe5b70c166e in term 0.
I20260925 14:52:57.389737 13361 leader_election.cc:302] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a81e57952f416cbaa86fe5b70c166e, c719d6c0c08944b59997a66b01c38e9e; no voters:
I20260925 14:52:57.389882 13642 raft_consensus.cc:2781] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:57.389941 13642 raft_consensus.cc:482] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:57.389971 13642 raft_consensus.cc:3037] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.390900 13642 raft_consensus.cc:504] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.391038 13642 leader_election.cc:288] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [CANDIDATE]: Term 1 election: Requested vote from peers c719d6c0c08944b59997a66b01c38e9e (127.0.98.1:37773), 7e21dfdbc5464691a7197aa2ae939fc6 (127.0.98.3:34145)
I20260925 14:52:57.391444 13560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9406831ef82e4db1a900055491a35e3f" candidate_uuid: "41a81e57952f416cbaa86fe5b70c166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e21dfdbc5464691a7197aa2ae939fc6"
I20260925 14:52:57.391449 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9406831ef82e4db1a900055491a35e3f" candidate_uuid: "41a81e57952f416cbaa86fe5b70c166e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c719d6c0c08944b59997a66b01c38e9e"
I20260925 14:52:57.391532 13292 raft_consensus.cc:3037] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.391531 13560 raft_consensus.cc:3037] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.392352 13560 raft_consensus.cc:2454] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a81e57952f416cbaa86fe5b70c166e in term 1.
I20260925 14:52:57.392477 13292 raft_consensus.cc:2454] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a81e57952f416cbaa86fe5b70c166e in term 1.
I20260925 14:52:57.392580 13362 leader_election.cc:302] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a81e57952f416cbaa86fe5b70c166e, 7e21dfdbc5464691a7197aa2ae939fc6; no voters:
I20260925 14:52:57.392715 13642 raft_consensus.cc:2781] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:57.392877 13642 raft_consensus.cc:686] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 1 LEADER]: Becoming Leader. State: Replica: 41a81e57952f416cbaa86fe5b70c166e, State: Running, Role: LEADER
I20260925 14:52:57.392966 13642 consensus_queue.cc:237] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } }
I20260925 14:52:57.393743 13140 catalog_manager.cc:5495] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a81e57952f416cbaa86fe5b70c166e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "41a81e57952f416cbaa86fe5b70c166e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a81e57952f416cbaa86fe5b70c166e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:57.467335 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.463522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.98.2:40467)
0925 14:52:57.463563 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.463565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.463572 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:52:57.463613 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.463613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.463636 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.463702 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:57.463916 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.464692 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.466256 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.466527 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.466583 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.466799 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:57.466932 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:57.467007 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.467235 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.467257 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:52:57.469623 13292 raft_consensus.cc:1261] T 9406831ef82e4db1a900055491a35e3f P c719d6c0c08944b59997a66b01c38e9e [term 1 FOLLOWER]: Refusing update from remote peer 41a81e57952f416cbaa86fe5b70c166e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.469623 13560 raft_consensus.cc:1261] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 1 FOLLOWER]: Refusing update from remote peer 41a81e57952f416cbaa86fe5b70c166e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.469961 13653 consensus_queue.cc:1035] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e21dfdbc5464691a7197aa2ae939fc6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.470065 13642 consensus_queue.cc:1035] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c719d6c0c08944b59997a66b01c38e9e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.472698 13658 mvcc.cc:205] Tried to move back new op lower bound from 7333265315712212992 to 7333265315402211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333265315712212992}]
I20260925 14:52:57.474205 13660 mvcc.cc:205] Tried to move back new op lower bound from 7333265315712212992 to 7333265315402211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333265315712212992}]
I20260925 14:52:57.476146 13661 mvcc.cc:205] Tried to move back new op lower bound from 7333265315712212992 to 7333265315402211328. Current Snapshot: MvccSnapshot[applied={T|T < 7333265315712212992 or (T in {7333265315712212992})}]
I20260925 14:52:57.476311 13139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.476414 13139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.477221 13139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d6a6f6194384ea0ac85f1d792b16bb1: Sending DeleteTablet for 3 replicas of tablet 9406831ef82e4db1a900055491a35e3f
I20260925 14:52:57.477592 13272 tablet_service.cc:1514] Processing DeleteTablet for tablet 9406831ef82e4db1a900055491a35e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50332
I20260925 14:52:57.477586 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet 9406831ef82e4db1a900055491a35e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34052
I20260925 14:52:57.477905 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet 9406831ef82e4db1a900055491a35e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41814
I20260925 14:52:57.478117 13674 tablet_replica.cc:330] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: stopping tablet replica
I20260925 14:52:57.478111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13178
I20260925 14:52:57.478976 13674 raft_consensus.cc:2229] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:57.479153 13674 raft_consensus.cc:2258] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.479664 13674 ts_tablet_manager.cc:1905] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.481755 13674 ts_tablet_manager.cc:1918] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.481849 13674 log.cc:1196] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/9406831ef82e4db1a900055491a35e3f
I20260925 14:52:57.482084 13674 ts_tablet_manager.cc:1939] T 9406831ef82e4db1a900055491a35e3f P 7e21dfdbc5464691a7197aa2ae939fc6: Deleting consensus metadata
I20260925 14:52:57.482295 13672 tablet_replica.cc:330] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: stopping tablet replica
I20260925 14:52:57.482427 13672 raft_consensus.cc:2229] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:57.482544 13672 raft_consensus.cc:2258] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.482972 13127 catalog_manager.cc:4841] TS 7e21dfdbc5464691a7197aa2ae939fc6 (127.0.98.3:34145): tablet 9406831ef82e4db1a900055491a35e3f (table test-table [id=8e1cd6c025f7480c9cb56925d642b9da]) successfully deleted
W20260925 14:52:57.485496 13126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37773 (error 108)
I20260925 14:52:57.485982 13672 ts_tablet_manager.cc:1905] T 9406831ef82e4db1a900055491a35e3f P 41a81e57952f416cbaa86fe5b70c166e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.486254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13342
I20260925 14:52:57.486811 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.486250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37773 (local address 127.0.0.1:50348)
0925 14:52:57.486726 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.486771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
W20260925 14:52:57.487466 13126 catalog_manager.cc:4584] TS c719d6c0c08944b59997a66b01c38e9e (127.0.98.1:37773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9406831ef82e4db1a900055491a35e3f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37773: connect: Connection refused (error 111)
I20260925 14:52:57.490597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13475
I20260925 14:52:57.490698 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.490584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40467 (local address 127.0.0.1:34064)
0925 14:52:57.490636 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.490665 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:52:57.494670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13107
I20260925 14:52:57.500002 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:52:57.845034 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:52:57.845127 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:52:58.189908 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:52:58.190 UTC [12174] LOG: received smart shutdown request
2026-09-25 14:52:58.193 UTC [12174] LOG: background worker "logical replication launcher" (PID 12183) exited with exit code 1
2026-09-25 14:52:58.193 UTC [12178] LOG: shutting down
2026-09-25 14:52:58.357 UTC [12174] LOG: database system is shut down
2026-09-25T14:52:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49147 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:58 dist-test-slave-gpc1 krb5kdc[13684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:58 dist-test-slave-gpc1 krb5kdc[13684](info): set up 2 sockets
Sep 25 14:52:58 dist-test-slave-gpc1 krb5kdc[13684](info): commencing operation
krb5kdc: starting...
W20260925 14:52:59.498196 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:59.616492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38179
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:59.697710 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.697932 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.697959 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.699146 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:59.699187 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.699201 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.699213 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:59.699225 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:59.700824 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:59.701088 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.701323 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.706244 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.706223 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.706287 13697 server_base.cc:1034] running on GCE node
W20260925 14:52:59.706210 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.706763 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.707106 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.708278 13697 hybrid_clock.cc:648] HybridClock initialized: now 1790347979708265 us; error 44 us; skew 500 ppm
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:59.720561 13697 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:59.721118 13697 webserver.cc:460] Webserver started at http://127.0.98.62:35407/ using document root <none> and password file <none>
I20260925 14:52:59.721369 13697 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.721439 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.721586 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.722743 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "1172652a91364f49bd785c6cadc9893c"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "1bd99a6d11a8549386705b1f60d87014"
server_key_iv: "cb807c382ccbd18fcbd4c7c7d0279170"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.723098 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "1172652a91364f49bd785c6cadc9893c"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "1bd99a6d11a8549386705b1f60d87014"
server_key_iv: "cb807c382ccbd18fcbd4c7c7d0279170"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.724418 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:59.725227 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.725447 13697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.725557 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "1172652a91364f49bd785c6cadc9893c"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "1bd99a6d11a8549386705b1f60d87014"
server_key_iv: "cb807c382ccbd18fcbd4c7c7d0279170"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.725644 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.742110 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.744006 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.744172 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.748314 13697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38179
I20260925 14:52:59.748349 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38179 every 8 connection(s)
I20260925 14:52:59.748751 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:59.749620 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.751199 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:52:59.751418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13697
I20260925 14:52:59.751516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:59.751715 392 external_mini_cluster.cc:1428] Setting key 31f3b0473b827eb9ac5a71354af25a3e
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 14:52:59.761477 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.751983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.0.98.62:38179)
0925 14:52:59.752216 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.752224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.752303 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:52:59.752868 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.752870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.752890 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.753001 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.753555 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.754429 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.755332 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.755661 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.755782 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.757865 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.757879 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.757879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.757885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.759817 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.760237 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.760240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.760240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.760321 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.760592 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.760593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.760594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.760805 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.760898 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.761097 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.761237 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260925 14:52:59.761477 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.751983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38179 (local address 127.0.0.1:45236)
0925 14:52:59.752169 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.752215 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.752705 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.753008 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.753010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.753297 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.753453 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.753458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.754549 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.754550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.755232 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.755234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.755751 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.755752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.755815 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.756392 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.756401 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.757767 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.759941 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.759947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.759949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.760142 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.760407 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.760407 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.760407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.760522 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.760876 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.760881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.760963 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.761072 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.761303 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":93,"thread_start_us":86,"threads_started":1}
I20260925 14:52:59.762530 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:52:59.762921 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.762778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48334)
0925 14:52:59.762833 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.762887 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:52:59.763108 13713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
I20260925 14:52:59.763566 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.763454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48336)
0925 14:52:59.763504 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.763534 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:52:59.763695 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:52:59.776829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41959
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:59.812958 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } attempt: 1
I20260925 14:52:59.813553 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.813377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48338)
0925 14:52:59.813450 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.813504 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:52:59.814006 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.813914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48344)
0925 14:52:59.813958 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.813986 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:52:59.814152 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
W20260925 14:52:59.859233 13772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.859442 13772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.859464 13772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.860598 13772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:59.860643 13772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.860657 13772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.860669 13772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:59.860682 13772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:59.862257 13772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:52:59.862542 13772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.862780 13772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.867668 13780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.867832 13772 server_base.cc:1034] running on GCE node
W20260925 14:52:59.867668 13777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.867690 13778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.868185 13772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.868474 13772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.869673 13772 hybrid_clock.cc:648] HybridClock initialized: now 1790347979869672 us; error 42 us; skew 500 ppm
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:59.872272 13772 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:52:59.872751 13772 webserver.cc:460] Webserver started at http://127.0.98.61:39269/ using document root <none> and password file <none>
I20260925 14:52:59.872973 13772 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.873042 13772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.873194 13772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.874265 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data/instance:
uuid: "7e2ce8ffc60f4c3883ea530678e22e41"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "34b0cfdc8b67f534b15352f0d2e87681"
server_key_iv: "eb45492528c925325aa0996d22ac5644"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.874704 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal/instance:
uuid: "7e2ce8ffc60f4c3883ea530678e22e41"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "34b0cfdc8b67f534b15352f0d2e87681"
server_key_iv: "eb45492528c925325aa0996d22ac5644"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.876029 13772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.876881 13787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.877099 13772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.877209 13772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "7e2ce8ffc60f4c3883ea530678e22e41"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-gpc1"
server_key: "34b0cfdc8b67f534b15352f0d2e87681"
server_key_iv: "eb45492528c925325aa0996d22ac5644"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.877285 13772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.882328 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } attempt: 2
I20260925 14:52:59.882853 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.882726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48360)
0925 14:52:59.882784 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.882823 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:52:59.883219 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.883118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48370)
0925 14:52:59.883166 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.883191 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:52:59.883308 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41959: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41959: connect: Connection refused (error 111)
I20260925 14:52:59.888136 13772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.890012 13772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.890172 13772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.893958 13772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41959
I20260925 14:52:59.894019 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41959 every 8 connection(s)
I20260925 14:52:59.894441 13772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:52:59.895318 13840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.896854 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:52:59.901701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13772
I20260925 14:52:59.901852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-1/wal/instance
I20260925 14:52:59.902045 392 external_mini_cluster.cc:1428] Setting key 1e9ae5f6a14ddf1e9b7978daf8c25cab
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:59.906785 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.897566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.0.98.62:38179)
0925 14:52:59.897610 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.897612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.897868 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:52:59.898381 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.898382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.898397 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.898462 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.899086 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.899570 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.900565 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.900783 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.900877 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.903478 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.903480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.903480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.903490 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.905267 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.905671 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.905673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.905673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.905740 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.906054 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.906055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.906055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.906211 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.906287 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.906578 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.906673 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:52:59.906812 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.897539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38179 (local address 127.0.0.1:45248)
0925 14:52:59.897732 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.897767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.898257 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.898540 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.898543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.898795 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.898981 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.898986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.899676 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.899678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.900461 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.900463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.900885 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.900887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.901009 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.901462 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.901470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.903331 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.905362 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.905368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.905369 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.905591 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.905811 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.905812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.905812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.905975 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.906281 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.906286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.906448 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.906533 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.906657 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260925 14:52:59.907612 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:52:59.916064 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.907897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48382)
0925 14:52:59.907949 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.907961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.908006 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.908255 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.908256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.908397 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.908438 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.908440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.909304 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.909305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.909795 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.909796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.910288 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.910289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.910377 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.910735 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.910739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.913123 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.914931 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.914932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.914932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.915135 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.915385 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.915385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.915386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.915445 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.915722 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.915723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.915759 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.915848 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.915957 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:52:59.916195 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.907975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.0.98.61:41959)
0925 14:52:59.908123 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.908127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.908141 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:59.908160 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.908162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.908173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.908256 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.908520 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.909213 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.909873 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.910202 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.910288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.913204 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.913213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.913213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.913221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.914845 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.915245 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.915249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.915249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.915315 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.915509 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.915510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.915510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.915651 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.915744 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.915881 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.916012 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260925 14:52:59.916705 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:52:59.917058 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.916946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56478)
0925 14:52:59.916997 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.917040 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:52:59.917178 13790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
I20260925 14:52:59.917584 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.917490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56484)
0925 14:52:59.917539 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.917562 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:52:59.917691 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44189: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:52:59.927040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44189
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:59.966925 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attempt: 1
I20260925 14:52:59.967491 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.967329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56488)
0925 14:52:59.967406 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.967455 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:52:59.967926 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.967821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56490)
0925 14:52:59.967870 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.967898 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:52:59.968065 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44189: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
I20260925 14:52:59.974412 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } attempt: 3
Sep 25 14:52:59 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:52:59.981698 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.974739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48386)
0925 14:52:59.974794 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.974806 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.974860 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.974977 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.974978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.975142 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.975229 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.975231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.975926 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.975927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.976441 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.976443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.976823 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.976824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.976909 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.977405 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.977410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.978650 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.980598 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.980599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.980600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.980791 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.981070 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.981071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.981071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.981131 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.981443 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.981444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.981483 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.981535 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.981613 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:52:59.981763 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.974798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.0.98.61:41959)
0925 14:52:59.974862 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.974864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.974886 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:52:59.974915 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.974916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.974927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.974974 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.975324 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.975836 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.976524 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.976749 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.976825 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.978734 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.978735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.978736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.978743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.980519 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.980930 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.980931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.980931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.980993 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.981196 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.981197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.981198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.981375 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.981447 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.981584 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.981668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:52:59.982156 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:52:59.982503 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.982413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56494)
0925 14:52:59.982447 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.982479 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:52:59.982806 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.982717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56498)
0925 14:52:59.982763 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.982788 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:52:59.982904 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44189: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
W20260925 14:53:00.010831 13847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:00.011049 13847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:00.011072 13847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:00.012161 13847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:00.012199 13847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:00.012213 13847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:00.012226 13847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:00.012238 13847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:00.013695 13847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38179,127.0.98.61:41959,127.0.98.60:44189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:00.013891 13847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:00.014128 13847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:00.014071 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attempt: 1
I20260925 14:53:00.014617 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.014464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56504)
0925 14:53:00.014533 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.014581 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:53:00.015090 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.014915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56510)
0925 14:53:00.014964 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.014998 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:53:00.015276 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44189: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
W20260925 14:53:00.019838 13852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.019878 13853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.019878 13855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:00.020179 13847 server_base.cc:1034] running on GCE node
I20260925 14:53:00.020396 13847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:00.020676 13847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:00.021831 13847 hybrid_clock.cc:648] HybridClock initialized: now 1790347980021819 us; error 28 us; skew 500 ppm
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:00.024384 13847 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:53:00.024852 13847 webserver.cc:460] Webserver started at http://127.0.98.60:44365/ using document root <none> and password file <none>
I20260925 14:53:00.025079 13847 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:00.025125 13847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:00.025223 13847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:00.026257 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data/instance:
uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-gpc1"
server_key: "c4cbaad7ee1540fed125c25448055b3d"
server_key_iv: "4abd32bf40c38baa846344cff4fbf7e3"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.026634 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal/instance:
uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-gpc1"
server_key: "c4cbaad7ee1540fed125c25448055b3d"
server_key_iv: "4abd32bf40c38baa846344cff4fbf7e3"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.027874 13847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:00.028663 13862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.028824 13847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:00.028910 13847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-gpc1"
server_key: "c4cbaad7ee1540fed125c25448055b3d"
server_key_iv: "4abd32bf40c38baa846344cff4fbf7e3"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.028963 13847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:00.036237 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attempt: 2
I20260925 14:53:00.036671 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.036544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56526)
0925 14:53:00.036605 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.036642 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:53:00.037088 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.036975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56542)
0925 14:53:00.037045 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.037071 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:53:00.037199 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44189: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44189: connect: Connection refused (error 111)
I20260925 14:53:00.048840 13847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:00.050833 13847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:00.050948 13847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:00.054819 13847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44189
I20260925 14:53:00.054890 13914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44189 every 8 connection(s)
I20260925 14:53:00.055243 13847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:53:00.056185 13915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:00.057693 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:00.061765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 13847
I20260925 14:53:00.061880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347825060285-392-0/minicluster-data/master-2/wal/instance
I20260925 14:53:00.062085 392 external_mini_cluster.cc:1428] Setting key eee180fdc43f6ad4fb0fe87e622f7117
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:00.067591 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.058417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.0.98.62:38179)
0925 14:53:00.058486 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.058488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.058676 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:53:00.059098 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.059099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.059108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.059178 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.059814 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.060306 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.061274 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.061462 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.061522 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.064265 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.064266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.064267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.064274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.065964 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.066465 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.066467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.066467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.066532 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.066880 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.066881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.066881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.066992 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.067042 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.067443 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.067516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:53:00.067836 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.058383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38179 (local address 127.0.0.1:45256)
0925 14:53:00.058544 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.058586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.059012 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.059250 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.059253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.059513 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.059712 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.059716 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.060388 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.060389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.061182 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.061184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.061593 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.061595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.061765 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.062446 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.062456 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.064178 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.066045 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.066051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.066053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.066386 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.066642 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.066643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.066643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.066799 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.067058 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.067065 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.067250 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.067374 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.067539 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:00.068738 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:00.071854 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.062545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.0.98.60:44189)
0925 14:53:00.062737 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.062741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.062789 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:53:00.062864 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.062866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.062883 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.062949 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.063621 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.064694 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.065602 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.066107 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.066166 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.068897 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.068906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.068907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.068916 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.070739 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.071063 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.071067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.071067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.071130 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.071326 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.071326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.071326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.071451 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.071635 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.071696 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.071767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:00.076303 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.069052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41959 (local address 127.0.0.1:48394)
0925 14:53:00.069100 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.069110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.069145 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.069286 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.069287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.069487 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.069549 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.069551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.070307 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.070308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.071057 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.071059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.071500 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.071501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.071572 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.071984 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.071989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.073250 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.075299 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.075300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.075300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.075456 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.075678 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.075679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.075679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.075738 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.075987 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.075988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.076029 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.076112 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.076210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:53:00.076301 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.069105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.0.98.61:41959)
0925 14:53:00.069147 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.069151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.069186 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:00.069211 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.069212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.069223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.069274 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.069633 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.070231 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.071155 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.071433 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.071484 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.073329 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.073331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.073331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.073339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.075232 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.075539 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.075540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.075541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.075601 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.075803 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.075804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.075804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.075919 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.075996 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.076138 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.076211 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:00.076653 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:00.080464 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.072778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.62:38179)
0925 14:53:00.072826 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.072828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.072836 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:00.072856 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.072857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.072866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.072913 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.073509 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.073959 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.074975 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.075155 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.075224 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.077551 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.077552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.077553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.077560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.079019 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.079387 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.079388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.079388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.079437 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.079898 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.079899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.079899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.079998 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.080222 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.080281 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.080369 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:00.083940 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.076896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56558)
0925 14:53:00.076990 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.077007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.077045 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.077182 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.077182 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.077335 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.077364 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.077366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.078280 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.078281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.078855 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.078857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.079303 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.079304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.079367 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.079757 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.079762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.081335 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.082991 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.082992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.082992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.083143 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.083327 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.083328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.083328 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.083375 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.083632 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.083633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.083669 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.083750 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.083844 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 14:53:00.084214 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.076954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.0.98.60:44189)
0925 14:53:00.076989 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.076991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.077036 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:53:00.077082 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.077083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.077093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.077148 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.077437 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.078171 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.078938 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.079232 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.079281 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.081395 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.081396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.081396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.081403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.082916 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.083210 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.083211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.083211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.083267 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.083448 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.083449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.083449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.083560 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.083633 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.083784 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.084164 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":300,"server-negotiator.queue_time_us":19}
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:00.085605 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: Bootstrap starting.
I20260925 14:53:00.086614 13915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.086956 13915 log.cc:824] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.087786 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: No bootstrap required, opened a new log
I20260925 14:53:00.088598 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.081147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48398 (local address 127.0.98.61:41959)
0925 14:53:00.081207 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.081209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.081217 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:00.081241 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.081242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.081251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.081305 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.081921 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.082546 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.083409 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.083613 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.083687 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.085936 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.085938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.085938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.085946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.087505 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.087844 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.087845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.087846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.087903 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.088089 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.088090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.088090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.088205 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.088394 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.088430 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.088503 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:53:00.089699 13915 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.089851 13915 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.089917 13915 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d69ab9fb614fa2b99d9c9ae3cec636, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.090103 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:00.090068 13915 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.090422 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.090426 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attempt: 2
I20260925 14:53:00.090469 13915 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:00.090547 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:00.092625 13847 master_runner.cc:186] Error getting master registration for 127.0.98.62:38179: OK, instance_id { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" instance_seqno: 1790347979747129 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:00.095709 13847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 14:53:00.097884 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:00.098014 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:00.098887 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.090647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56572)
0925 14:53:00.090749 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.090760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.090798 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.090931 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.090932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.091091 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.091140 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.091142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.092045 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.092046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.092749 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.092751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.093213 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.093214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.093267 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.093697 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.093703 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.095521 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.097673 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.097674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.097675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.097947 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.098160 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.098161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.098162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.098224 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.098562 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.098563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.098606 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.098672 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.098774 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 14:53:00.099241 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.090747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.0.98.60:44189)
0925 14:53:00.090783 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.090785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.090794 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:00.090856 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.090857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.090866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.090924 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.091216 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.091943 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.092833 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.093106 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.093166 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.095625 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.095627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.095627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.095634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.097581 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.098028 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.098029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.098029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.098087 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.098291 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.098292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.098292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.098474 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.098535 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.099076 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.099161 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:00.099992 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.091797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.98.61:41959)
0925 14:53:00.091833 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.091835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.091941 (+ 106us) server_negotiation.cc:407] Connection header received
0925 14:53:00.092043 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.092044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.092054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.092083 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.092856 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.093559 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.094544 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.094846 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.094883 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.096418 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.096420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.096420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.096428 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.098700 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.099138 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.099139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.099140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.099192 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.099413 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.099414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.099415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.099546 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.099630 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.099825 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.099900 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:00.100667 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.091667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.98.62:38179)
0925 14:53:00.091705 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.091707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.092587 (+ 880us) server_negotiation.cc:407] Connection header received
0925 14:53:00.092680 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.092681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.092691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.092722 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.093403 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.094070 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.095026 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.095333 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.095374 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.096756 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.096758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.096758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.096766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.098588 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.099576 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.099577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.099578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.099635 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.099848 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.099849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.099849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.099968 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.100067 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.100494 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.100565 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:00.100924 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: Bootstrap starting.
I20260925 14:53:00.101641 13765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.101939 13765 log.cc:824] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.103130 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: No bootstrap required, opened a new log
I20260925 14:53:00.104673 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.091989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.98.60:44189)
0925 14:53:00.092253 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.092255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.095826 (+ 3571us) server_negotiation.cc:407] Connection header received
0925 14:53:00.095850 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.095851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.095863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.095928 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.097607 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.099280 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.100235 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.100427 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.100503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.101668 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.101669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.101670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.101676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.103424 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.103805 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.103806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.103807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.103869 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.104084 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.104085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.104085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.104216 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.104325 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.104463 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.104545 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260925 14:53:00.105192 13765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.105324 13765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.105371 13765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1172652a91364f49bd785c6cadc9893c, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.105482 13765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.105726 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.105790 13765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:00.105804 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:53:00.107493 13954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:00.107548 13954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:00.108434 13697 master_runner.cc:186] Error getting master registration for 127.0.98.61:41959: OK, instance_id { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" instance_seqno: 1790347979892955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:00.110905 13697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:53:00.128387 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } attempt: 3
Sep 25 14:53:00 dist-test-slave-gpc1 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:00.136911 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.128674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44189 (local address 127.0.0.1:56584)
0925 14:53:00.128725 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.128736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.128773 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.128921 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.128922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.129131 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.129184 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.129186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.130173 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.130174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.131037 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.131040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.131542 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.131544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.131623 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.132185 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:00.132191 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.133739 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.135707 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.135709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.135709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.135951 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.136202 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:00.136203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:00.136204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:00.136269 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:00.136540 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.136541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.136593 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.136686 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.136803 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:53:00.136914 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.128736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.98.60:44189)
0925 14:53:00.128787 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.128789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.128797 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:00.128822 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.128822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.128841 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.128903 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.129299 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.130080 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.131123 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.131453 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.131526 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.133836 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.133838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.133838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:00.133846 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.135573 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.136029 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.136030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.136031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.136113 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:00.136338 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:00.136338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:00.136338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:00.136469 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.136534 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.136724 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.136815 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:53:00.138626 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: Bootstrap starting.
I20260925 14:53:00.139369 13840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.139657 13840 log.cc:824] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.141054 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: No bootstrap required, opened a new log
I20260925 14:53:00.143040 13840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.143179 13840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.143213 13840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2ce8ffc60f4c3883ea530678e22e41, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.143390 13840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.143662 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.143713 13840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:00.143821 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:53:00.145582 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:00.145689 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:00.147872 13772 master_runner.cc:263] No leader master found from master 7e2ce8ffc60f4c3883ea530678e22e41
I20260925 14:53:00.147921 13772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:53:00.185803 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:00.185941 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.186216 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1172652a91364f49bd785c6cadc9893c (127.0.98.62:38179), c4d69ab9fb614fa2b99d9c9ae3cec636 (127.0.98.60:44189)
I20260925 14:53:00.186493 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1172652a91364f49bd785c6cadc9893c" is_pre_election: true
I20260925 14:53:00.186636 13739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e2ce8ffc60f4c3883ea530678e22e41 in term 0.
I20260925 14:53:00.187065 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" is_pre_election: true
I20260925 14:53:00.187208 13889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e2ce8ffc60f4c3883ea530678e22e41 in term 0.
I20260925 14:53:00.187590 13790 leader_election.cc:302] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e2ce8ffc60f4c3883ea530678e22e41, c4d69ab9fb614fa2b99d9c9ae3cec636; no voters:
I20260925 14:53:00.187839 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:00.187901 13957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:00.187925 13957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.189190 13957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.189306 13957 leader_election.cc:288] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [CANDIDATE]: Term 1 election: Requested vote from peers 1172652a91364f49bd785c6cadc9893c (127.0.98.62:38179), c4d69ab9fb614fa2b99d9c9ae3cec636 (127.0.98.60:44189)
I20260925 14:53:00.189603 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1172652a91364f49bd785c6cadc9893c"
I20260925 14:53:00.189685 13739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.190538 13889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636"
I20260925 14:53:00.190619 13889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.191542 13889 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e2ce8ffc60f4c3883ea530678e22e41 in term 1.
I20260925 14:53:00.191969 13790 leader_election.cc:302] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e2ce8ffc60f4c3883ea530678e22e41, c4d69ab9fb614fa2b99d9c9ae3cec636; no voters:
I20260925 14:53:00.192361 13739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e2ce8ffc60f4c3883ea530678e22e41 in term 1.
I20260925 14:53:00.192586 13957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:00.192770 13957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [term 1 LEADER]: Becoming Leader. State: Replica: 7e2ce8ffc60f4c3883ea530678e22e41, State: Running, Role: LEADER
I20260925 14:53:00.192862 13957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } }
I20260925 14:53:00.193251 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2ce8ffc60f4c3883ea530678e22e41. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.193321 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:00.193611 13972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:00.193854 13972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:00.195979 13889 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [term 1 FOLLOWER]: Refusing update from remote peer 7e2ce8ffc60f4c3883ea530678e22e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:00.196251 13739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [term 1 FOLLOWER]: Refusing update from remote peer 7e2ce8ffc60f4c3883ea530678e22e41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:00.196686 13957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:00.196862 13957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:00.199085 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2ce8ffc60f4c3883ea530678e22e41. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.199170 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:00.200686 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2ce8ffc60f4c3883ea530678e22e41. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.200785 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:00.202644 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.202742 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:00.203140 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.203218 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:00.203471 13973 mvcc.cc:205] Tried to move back new op lower bound from 7333265326880002048 to 7333265326870622208. Current Snapshot: MvccSnapshot[applied={T|T < 7333265326880002048}]
I20260925 14:53:00.203967 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.204046 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:00.204445 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1172652a91364f49bd785c6cadc9893c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38179 } } peers { permanent_uuid: "7e2ce8ffc60f4c3883ea530678e22e41" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41959 } } peers { permanent_uuid: "c4d69ab9fb614fa2b99d9c9ae3cec636" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44189 } } }
I20260925 14:53:00.204526 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:00.205202 13974 mvcc.cc:205] Tried to move back new op lower bound from 7333265326880002048 to 7333265326870622208. Current Snapshot: MvccSnapshot[applied={T|T < 7333265326880002048}]
I20260925 14:53:00.205767 13972 catalog_manager.cc:1347] Generated new cluster ID: 17e711a373444066aebff7759234d1d7
I20260925 14:53:00.205761 13976 mvcc.cc:205] Tried to move back new op lower bound from 7333265326880002048 to 7333265326870622208. Current Snapshot: MvccSnapshot[applied={T|T < 7333265326880002048}]
I20260925 14:53:00.205816 13972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:00.244374 13972 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:00.245234 13972 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:00.253990 13972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e2ce8ffc60f4c3883ea530678e22e41: Generated new TSK 0
I20260925 14:53:00.254266 13972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:01.098618 13938 catalog_manager.cc:1259] Loaded cluster ID: 17e711a373444066aebff7759234d1d7
I20260925 14:53:01.098721 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: loading cluster ID for follower catalog manager: success
I20260925 14:53:01.099574 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: acquiring CA information for follower catalog manager: success
I20260925 14:53:01.099896 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4d69ab9fb614fa2b99d9c9ae3cec636: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:01.107951 13954 catalog_manager.cc:1259] Loaded cluster ID: 17e711a373444066aebff7759234d1d7
I20260925 14:53:01.108008 13954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: loading cluster ID for follower catalog manager: success
I20260925 14:53:01.108779 13954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: acquiring CA information for follower catalog manager: success
I20260925 14:53:01.109036 13954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1172652a91364f49bd785c6cadc9893c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:01.306239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13697
I20260925 14:53:01.311688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13772
I20260925 14:53:01.316663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 13847
2026-09-25T14:53:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2850 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:01 dist-test-slave-gpc1 krb5kdc[13987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:01 dist-test-slave-gpc1 krb5kdc[13987](info): set up 2 sockets
Sep 25 14:53:01 dist-test-slave-gpc1 krb5kdc[13987](info): commencing operation
krb5kdc: starting...
W20260925 14:53:02.354516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:02.471952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:02.554595 14000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:02.554802 14000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:02.554826 14000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:02.556030 14000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:02.556074 14000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:02.556088 14000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:02.556128 14000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:02.556155 14000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:02.557696 14000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:02.557945 14000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:02.558183 14000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:02.563040 14000 server_base.cc:1034] running on GCE node
W20260925 14:53:02.563009 14005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:02.563108 14006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:02.563012 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:02.563566 14000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:02.563868 14000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:02.565044 14000 hybrid_clock.cc:648] HybridClock initialized: now 1790347982565037 us; error 36 us; skew 500 ppm
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[13987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:02.567550 14000 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:02.568063 14000 webserver.cc:460] Webserver started at http://127.0.98.62:33409/ using document root <none> and password file <none>
I20260925 14:53:02.568260 14000 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:02.568300 14000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:02.568385 14000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:02.569379 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "14301375803a4064990b197f89206bec"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-gpc1"
server_key: "1097a446d6c3c7fede295e8f849c94fe"
server_key_iv: "d673ccaa9a104ad885cc7e1d5e7a183a"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.569674 14000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "14301375803a4064990b197f89206bec"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-gpc1"
server_key: "1097a446d6c3c7fede295e8f849c94fe"
server_key_iv: "d673ccaa9a104ad885cc7e1d5e7a183a"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.570972 14000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:02.571667 14015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:02.571843 14000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:02.571907 14000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "14301375803a4064990b197f89206bec"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-gpc1"
server_key: "1097a446d6c3c7fede295e8f849c94fe"
server_key_iv: "d673ccaa9a104ad885cc7e1d5e7a183a"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.571957 14000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:02.599906 14000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:02.601907 14000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:02.602025 14000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:02.606218 14000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41585
I20260925 14:53:02.606287 14067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41585 every 8 connection(s)
I20260925 14:53:02.606659 14000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:02.607534 14068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:02.608254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14000
I20260925 14:53:02.608372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:02.608553 392 external_mini_cluster.cc:1428] Setting key 3abd8e6cfce9edd4f40374a5aeb6bed4
I20260925 14:53:02.610793 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: Bootstrap starting.
I20260925 14:53:02.611506 14068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:02.611795 14068 log.cc:824] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:02.612521 14068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: No bootstrap required, opened a new log
I20260925 14:53:02.614883 14068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260925 14:53:02.615017 14068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:02.615039 14068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14301375803a4064990b197f89206bec, State: Initialized, Role: FOLLOWER
I20260925 14:53:02.615206 14068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260925 14:53:02.615306 14068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:02.615348 14068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:02.615399 14068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 0 FOLLOWER]: Advancing to term 1
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[13987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:02.615986 14068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260925 14:53:02.616111 14068 leader_election.cc:302] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14301375803a4064990b197f89206bec; no voters:
I20260925 14:53:02.616299 14068 leader_election.cc:288] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:02.616343 14073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:02.616567 14068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:02.616544 14073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [term 1 LEADER]: Becoming Leader. State: Replica: 14301375803a4064990b197f89206bec, State: Running, Role: LEADER
I20260925 14:53:02.618032 14073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } }
I20260925 14:53:02.618695 14075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14301375803a4064990b197f89206bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } }
I20260925 14:53:02.618784 14075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:02.618796 14074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14301375803a4064990b197f89206bec. Latest consensus state: current_term: 1 leader_uuid: "14301375803a4064990b197f89206bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14301375803a4064990b197f89206bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41585 } } }
I20260925 14:53:02.618889 14074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:02.619098 14087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:02.619159 14087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:02.619498 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:02.619805 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:02.620579 14071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:02.609106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.0.98.62:41585)
0925 14:53:02.609306 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:02.609315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:02.609350 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:02.609954 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:02.609963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:02.610029 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:02.610184 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:02.610846 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.612190 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.613018 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:02.613650 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:02.613759 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:02.616326 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:02.616330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:02.616330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:02.616344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:02.618706 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:02.619137 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:02.619140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:02.619141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:02.619238 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:02.619490 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:02.619491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:02.619492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:02.619819 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:02.619991 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:02.620239 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 14:53:02.620384 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260925 14:53:02.621575 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:02.626812 14088 catalog_manager.cc:1347] Generated new cluster ID: d38a64a56567490eafaa6d5936c24e86
I20260925 14:53:02.626878 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:02.632467 14088 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:02.633185 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:02.646386 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14301375803a4064990b197f89206bec: Generated new TSK 0
I20260925 14:53:02.646620 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:02.675863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14000
2026-09-25T14:53:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[14095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[14095](info): set up 2 sockets
Sep 25 14:53:02 dist-test-slave-gpc1 krb5kdc[14095](info): commencing operation
krb5kdc: starting...
W20260925 14:53:03.706492 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:03 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:03.823299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:03.904251 14108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:03.904457 14108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:03.904479 14108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:03.905616 14108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:03.905659 14108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:03.905673 14108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:03.905685 14108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:03.905697 14108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:03.907150 14108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:03.907449 14108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:03.907693 14108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:03.912508 14114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:03.912494 14113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:03.912494 14116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:03.912842 14108 server_base.cc:1034] running on GCE node
I20260925 14:53:03.913131 14108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:03.913437 14108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:03.914597 14108 hybrid_clock.cc:648] HybridClock initialized: now 1790347983914567 us; error 34 us; skew 500 ppm
Sep 25 14:53:03 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:03.923134 14108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:03.923712 14108 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260925 14:53:03.923955 14108 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:03.924009 14108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:03.924147 14108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:03.925269 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "384b9314247d48bdb56d0ec0682c75f8"
format_stamp: "Formatted at 2026-09-25 14:53:03 on dist-test-slave-gpc1"
server_key: "6620fd343d3e60e2e635cfe95d1f088c"
server_key_iv: "98a2233e2358fb864746edaa3f8b26ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:03.925657 14108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "384b9314247d48bdb56d0ec0682c75f8"
format_stamp: "Formatted at 2026-09-25 14:53:03 on dist-test-slave-gpc1"
server_key: "6620fd343d3e60e2e635cfe95d1f088c"
server_key_iv: "98a2233e2358fb864746edaa3f8b26ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:03.927199 14108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:03.928025 14123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:03.928269 14108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:03.928349 14108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "384b9314247d48bdb56d0ec0682c75f8"
format_stamp: "Formatted at 2026-09-25 14:53:03 on dist-test-slave-gpc1"
server_key: "6620fd343d3e60e2e635cfe95d1f088c"
server_key_iv: "98a2233e2358fb864746edaa3f8b26ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:03.928443 14108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:03.937602 14108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:03.939779 14108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:03.939940 14108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:03.944267 14108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38245
I20260925 14:53:03.944389 14175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38245 every 8 connection(s)
I20260925 14:53:03.944711 14108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:03.945778 14176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:03.948109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14108
I20260925 14:53:03.948235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:03.948460 392 external_mini_cluster.cc:1428] Setting key 4c0ad71e17144ac8cc1fe5c3773522a6
I20260925 14:53:03.948552 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap starting.
I20260925 14:53:03.949703 14176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:03.950067 14176 log.cc:824] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:03.950874 14176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: No bootstrap required, opened a new log
I20260925 14:53:03.952883 14176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:03.953066 14176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:03.953125 14176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Initialized, Role: FOLLOWER
I20260925 14:53:03.953313 14176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:03.953420 14176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:03.953485 14176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:03.953542 14176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:03.954334 14176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:03.954499 14176 leader_election.cc:302] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384b9314247d48bdb56d0ec0682c75f8; no voters:
I20260925 14:53:03.954731 14176 leader_election.cc:288] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:03.954752 14181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:03.954876 14181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 LEADER]: Becoming Leader. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Running, Role: LEADER
I20260925 14:53:03.955034 14176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:03.955018 14181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:03.955557 14183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384b9314247d48bdb56d0ec0682c75f8. Latest consensus state: current_term: 1 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:03.955657 14183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:53:03 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:03.956084 14182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:03.956175 14182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:03.956471 14189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:03.957019 14189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:03.958678 14189 catalog_manager.cc:1347] Generated new cluster ID: 4928e8ae22f44470a41fb4a8a019c5f4
I20260925 14:53:03.958734 14189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:03.960119 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.949061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.98.62:38245)
0925 14:53:03.949288 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.949301 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.949341 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:53:03.950039 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.950049 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.950118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.950291 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.951119 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.952033 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.952839 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.953355 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.953483 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.956418 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.956422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.956423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:03.956435 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.958457 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.958900 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.958903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.958904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.959004 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:03.959241 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:03.959242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:03.959242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:03.959517 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.959657 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.959813 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.959939 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260925 14:53:03.967553 14189 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:03.968263 14189 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:03.975721 14189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Generated new TSK 0
I20260925 14:53:03.975909 14189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:53:04.001824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:04.104529 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.104789 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.104832 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.105993 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.106057 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.106104 14202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:53:04.107654 14202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:04.107918 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.108153 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.113957 14210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.113946 14208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.113926 14207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.114146 14202 server_base.cc:1034] running on GCE node
I20260925 14:53:04.114473 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.114732 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.115896 14202 hybrid_clock.cc:648] HybridClock initialized: now 1790347984115881 us; error 28 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.118511 14202 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:53:04.119019 14202 webserver.cc:460] Webserver started at http://127.0.98.1:36601/ using document root <none> and password file <none>
I20260925 14:53:04.119254 14202 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.119325 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.119464 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:04.120543 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "6589127d13064be788c088a3f69fa096"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "bfe47555622a92e4e72a57c88a822e7c"
server_key_iv: "dd3b6bed5031ab8f91f03ff13bb26ee2"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.120918 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6589127d13064be788c088a3f69fa096"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "bfe47555622a92e4e72a57c88a822e7c"
server_key_iv: "dd3b6bed5031ab8f91f03ff13bb26ee2"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.122229 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:04.122983 14217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.123183 14202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:04.123278 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "6589127d13064be788c088a3f69fa096"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "bfe47555622a92e4e72a57c88a822e7c"
server_key_iv: "dd3b6bed5031ab8f91f03ff13bb26ee2"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.123360 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.137434 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.139856 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.140020 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.140257 14202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:04.140558 14202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:04.140609 14202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.140661 14202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:04.140692 14202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.147117 14202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42137
I20260925 14:53:04.147171 14330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42137 every 8 connection(s)
I20260925 14:53:04.147471 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:04.148015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14202
I20260925 14:53:04.148128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:04.148351 392 external_mini_cluster.cc:1428] Setting key 95ce5f7f4800b8cecd007de2a0a80456
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:53:04.158103 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.148523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56183 (local address 127.0.98.62:38245)
0925 14:53:04.148572 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.148575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.148587 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:04.149061 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.149062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.149072 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.149112 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.150455 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.151279 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.152259 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.152603 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.152683 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.154939 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.154941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.154941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.154950 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.156715 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.157121 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.157122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.157122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.157185 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.157433 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.157434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.157434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.157558 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.157634 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.157894 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.157987 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:04.158303 14333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.148215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:56183)
0925 14:53:04.148403 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.148451 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.148950 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.149199 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.149204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.150053 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:04.150329 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.150335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.151373 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.151374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.152170 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.152172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.152694 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.152695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.152797 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.153352 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:04.153360 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:04.154852 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:04.156790 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.156796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.156798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.157015 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.157252 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.157253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.157254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.157358 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.157624 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:04.157631 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:04.157764 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.157873 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.158070 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260925 14:53:04.158867 14331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:04.158969 14331 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:04.159168 14331 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:53:04.159705 14140 ts_manager.cc:194] Registered new tserver with Master: 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:04.160318 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:53:04.173691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:04.254695 14336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.254892 14336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.254915 14336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.256198 14336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.256242 14336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.256299 14336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:53:04.257722 14336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:04.257988 14336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.258222 14336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.263259 14344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.263319 14342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.263392 14341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.263650 14336 server_base.cc:1034] running on GCE node
I20260925 14:53:04.263846 14336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.264052 14336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.265214 14336 hybrid_clock.cc:648] HybridClock initialized: now 1790347984265187 us; error 39 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.267872 14336 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:53:04.268306 14336 webserver.cc:460] Webserver started at http://127.0.98.2:45167/ using document root <none> and password file <none>
I20260925 14:53:04.268553 14336 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.268621 14336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.268771 14336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:04.269814 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "3987721521474cf2b9df1a6b691ba25e"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "8ec16c349fb99fc6357455e677043818"
server_key_iv: "d865ff706448ce252d9bafc8594db3b9"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.270172 14336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3987721521474cf2b9df1a6b691ba25e"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "8ec16c349fb99fc6357455e677043818"
server_key_iv: "d865ff706448ce252d9bafc8594db3b9"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.271538 14336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:04.272238 14351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.272404 14336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:04.272492 14336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "3987721521474cf2b9df1a6b691ba25e"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "8ec16c349fb99fc6357455e677043818"
server_key_iv: "d865ff706448ce252d9bafc8594db3b9"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.272593 14336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.281911 14336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.284158 14336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.284310 14336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.284529 14336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:04.284808 14336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:04.284858 14336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.284898 14336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:04.284925 14336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.290480 14336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41617
I20260925 14:53:04.290557 14464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41617 every 8 connection(s)
I20260925 14:53:04.290870 14336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:04.298386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14336
I20260925 14:53:04.298516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:04.298743 392 external_mini_cluster.cc:1428] Setting key a4eb461eb593b5ec1f5e7fcc5d2e1232
I20260925 14:53:04.300443 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.291661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47877 (local address 127.0.98.62:38245)
0925 14:53:04.291764 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.291767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.291933 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:53:04.292477 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.292478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.292491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.292557 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.293479 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.293994 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.295137 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.295336 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.295437 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.297462 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.297463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.297464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.297470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.298948 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.299413 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.299414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.299415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.299484 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.299767 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.299768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.299768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.299864 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.299920 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.300297 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.300379 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 14:53:04.300987 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.291564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:47877)
0925 14:53:04.291798 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.291841 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.292371 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.292629 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.292633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.293192 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:04.293398 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.293406 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.294088 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.294089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.295047 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.295050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.295446 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.295447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.295573 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.296033 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:04.296041 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:04.297382 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:04.299041 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.299047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.299049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.299343 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.299553 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.299554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.299554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.299698 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.299925 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:04.299931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:04.300104 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.300226 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.300398 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":125,"threads_started":1}
I20260925 14:53:04.301404 14465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:04.301494 14465 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:04.301673 14465 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
I20260925 14:53:04.302045 14140 ts_manager.cc:194] Registered new tserver with Master: 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
I20260925 14:53:04.302450 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47877
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:53:04.322824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:04.406453 14470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.406670 14470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.406705 14470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.407852 14470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.407920 14470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.407961 14470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:04.409439 14470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:04.409704 14470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.409929 14470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.414918 14476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.414909 14478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.414909 14475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.415287 14470 server_base.cc:1034] running on GCE node
I20260925 14:53:04.415578 14470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.415819 14470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.416996 14470 hybrid_clock.cc:648] HybridClock initialized: now 1790347984416989 us; error 36 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.419574 14470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:53:04.419963 14470 webserver.cc:460] Webserver started at http://127.0.98.3:42557/ using document root <none> and password file <none>
I20260925 14:53:04.420207 14470 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.420253 14470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.420383 14470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:04.421458 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "3915efee0d08492c84356abdcdc1e4b1"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "7e87b9ef4e5bb3a60f12f61bf3cdd2fa"
server_key_iv: "60fba0c1ecebbdba48084aaf1622264e"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.421823 14470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3915efee0d08492c84356abdcdc1e4b1"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "7e87b9ef4e5bb3a60f12f61bf3cdd2fa"
server_key_iv: "60fba0c1ecebbdba48084aaf1622264e"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.423281 14470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:04.424043 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.424232 14470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:04.424340 14470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "3915efee0d08492c84356abdcdc1e4b1"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "7e87b9ef4e5bb3a60f12f61bf3cdd2fa"
server_key_iv: "60fba0c1ecebbdba48084aaf1622264e"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.424422 14470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.439520 14470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.441949 14470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.442121 14470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.442397 14470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:04.442708 14470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:04.442767 14470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.442811 14470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:04.442838 14470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.448776 14470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35043
I20260925 14:53:04.448817 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35043 every 8 connection(s)
I20260925 14:53:04.449172 14470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:04.457362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14470
I20260925 14:53:04.457501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:04.457688 392 external_mini_cluster.cc:1428] Setting key 54ad93c56471998c2538dc31d9e7f8d0
I20260925 14:53:04.458981 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.449912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60507 (local address 127.0.98.62:38245)
0925 14:53:04.449973 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.449975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.450205 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:53:04.450781 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.450782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.450792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.450848 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.451755 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.452258 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.453336 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.453544 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.453616 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.456003 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.456004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.456004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.456011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.457403 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.457880 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.457881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.457881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.457959 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.458279 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.458280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.458281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.458466 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.458547 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.458851 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.458919 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:53:04.459235 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.449895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:60507)
0925 14:53:04.450074 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.450119 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.450696 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.450902 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.450905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.451468 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:04.451665 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.451669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.452339 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.452339 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.453210 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.453212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.453627 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.453628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.453725 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.454243 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:04.454252 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:04.455921 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:04.457515 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.457522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.457524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.457803 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.458042 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.458042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.458042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.458208 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.458540 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:04.458548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:04.458713 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.458826 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.458953 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":145,"threads_started":1}
I20260925 14:53:04.459672 14599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:04.459781 14599 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:04.459955 14599 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
I20260925 14:53:04.460382 14140 ts_manager.cc:194] Registered new tserver with Master: 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043)
I20260925 14:53:04.460742 14140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60507
I20260925 14:53:04.469805 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:04.480545 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.471486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.0.98.62:38245)
0925 14:53:04.471538 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.471540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.472960 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 14:53:04.473095 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.473097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.473110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.473180 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.473897 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.474738 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.475865 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.476204 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.476274 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.477908 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.477909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.477910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.477920 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.479576 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.479888 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.479889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.479889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.479939 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.480071 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.480072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.480072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.480222 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.480294 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.480388 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.480454 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:04.482107 14140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:04.483719 14140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:04.490155 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.485877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35043 (local address 127.0.0.1:52418)
0925 14:53:04.486175 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.486218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.486288 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.486574 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.486576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.486579 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:04.486652 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.486654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.487858 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.487859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.489013 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.489015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.489860 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.489861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.489938 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.489951 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.490023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.490083 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":70,"threads_started":1}
I20260925 14:53:04.490170 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.485981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.0.98.3:35043)
0925 14:53:04.486404 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.486407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.486416 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:04.486453 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.486456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.486475 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.486571 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:04.486743 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.487723 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.489114 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.489769 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.489862 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.489923 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.490038 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.490079 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":92,"threads_started":1}
I20260925 14:53:04.490600 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.485877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.0.1:55042)
0925 14:53:04.486671 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.486681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.486894 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.487148 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.487149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.487152 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:04.487217 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.487218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.488457 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.488458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.489337 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.489339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.490320 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.490321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.490416 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.490421 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.490476 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.490522 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260925 14:53:04.490950 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.486089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.1:42137)
0925 14:53:04.486291 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.486296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.486736 (+ 440us) server_negotiation.cc:407] Connection header received
0925 14:53:04.486974 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.486977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.486994 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.487065 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:04.487315 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.488361 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.489429 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.490148 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.490224 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.490281 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.490724 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.490781 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260925 14:53:04.491096 14533 tablet_service.cc:1467] Processing CreateTablet for tablet d4b86394372148e8b808441f8175dccf (DEFAULT_TABLE table=test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:04.491480 14533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b86394372148e8b808441f8175dccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:04.491618 14265 tablet_service.cc:1467] Processing CreateTablet for tablet d4b86394372148e8b808441f8175dccf (DEFAULT_TABLE table=test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:04.491971 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b86394372148e8b808441f8175dccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:04.492224 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.486226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.0.98.2:41617)
0925 14:53:04.486408 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.486414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.486849 (+ 435us) server_negotiation.cc:407] Connection header received
0925 14:53:04.486909 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.486911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.486929 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.487023 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:04.487101 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.488215 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.489495 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.490165 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.491668 (+ 1503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.491738 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.492080 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.492141 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260925 14:53:04.492384 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.486097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41617 (local address 127.0.0.1:43128)
0925 14:53:04.486784 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.486793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.486844 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.487015 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.487016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.487018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:04.487057 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.487058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.488324 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.488325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.489426 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.489428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.491768 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.491769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.491867 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.491872 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.491926 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.491968 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":23,"thread_start_us":77,"threads_started":1}
I20260925 14:53:04.493139 14399 tablet_service.cc:1467] Processing CreateTablet for tablet d4b86394372148e8b808441f8175dccf (DEFAULT_TABLE table=test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:04.493474 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4b86394372148e8b808441f8175dccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:04.494331 14616 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Bootstrap starting.
I20260925 14:53:04.494975 14616 tablet_bootstrap.cc:654] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:04.495332 14617 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Bootstrap starting.
I20260925 14:53:04.495532 14616 log.cc:824] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.495954 14617 tablet_bootstrap.cc:654] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:04.496419 14616 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: No bootstrap required, opened a new log
I20260925 14:53:04.496481 14616 ts_tablet_manager.cc:1397] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:53:04.496485 14617 log.cc:824] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.496546 14619 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Bootstrap starting.
I20260925 14:53:04.497131 14619 tablet_bootstrap.cc:654] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:04.497682 14619 log.cc:824] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.498330 14619 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: No bootstrap required, opened a new log
I20260925 14:53:04.498445 14619 ts_tablet_manager.cc:1397] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:04.498278 14616 raft_consensus.cc:348] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.498482 14616 raft_consensus.cc:374] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:04.498529 14616 raft_consensus.cc:729] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3915efee0d08492c84356abdcdc1e4b1, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.498651 14616 consensus_queue.cc:260] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.498924 14616 ts_tablet_manager.cc:1428] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 14:53:04.498987 14599 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
I20260925 14:53:04.499559 14617 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: No bootstrap required, opened a new log
I20260925 14:53:04.499631 14617 ts_tablet_manager.cc:1397] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 14:53:04.500440 14619 raft_consensus.cc:348] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.500612 14619 raft_consensus.cc:374] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:04.500649 14619 raft_consensus.cc:729] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3987721521474cf2b9df1a6b691ba25e, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.500767 14619 consensus_queue.cc:260] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.501044 14619 ts_tablet_manager.cc:1428] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:53:04.501070 14465 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
I20260925 14:53:04.501592 14617 raft_consensus.cc:348] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.501791 14617 raft_consensus.cc:374] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:04.501843 14617 raft_consensus.cc:729] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6589127d13064be788c088a3f69fa096, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.501962 14617 consensus_queue.cc:260] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.502233 14617 ts_tablet_manager.cc:1428] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:04.502272 14331 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
W20260925 14:53:04.541460 14466 tablet.cc:2369] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:53:04.648206 14332 tablet.cc:2369] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:04.651371 14623 raft_consensus.cc:482] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:04.651487 14623 raft_consensus.cc:504] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.651902 14623 leader_election.cc:288] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:04.654958 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.652094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35043 (local address 127.0.98.2:46025)
0925 14:53:04.652165 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.652178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.652224 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.652453 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.652455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.652458 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:04.652559 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.652562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.653246 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.653247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.654041 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.654044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.654625 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.654782 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.654787 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.654838 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.654884 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":2}
I20260925 14:53:04.654990 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.652201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46025 (local address 127.0.98.3:35043)
0925 14:53:04.652312 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.652314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.652323 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:04.652352 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.652353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.652366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.652421 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:04.652641 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.653155 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654133 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.654545 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654605 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.654614 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.654887 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.654926 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 14:53:04.655023 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.652214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57323 (local address 127.0.98.1:42137)
0925 14:53:04.652276 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.652279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.652459 (+ 180us) server_negotiation.cc:407] Connection header received
0925 14:53:04.652489 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.652490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.652502 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.652558 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:04.652735 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.653175 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654176 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.654626 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654696 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.654705 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.654910 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.654951 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:53:04.655090 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.652100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.98.2:57323)
0925 14:53:04.652348 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.652357 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.652416 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.652573 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.652573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.652574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:04.652661 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.652663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.653286 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.653286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.654041 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.654043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.654704 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.654705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.654800 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.654804 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.654895 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.654934 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":19,"thread_start_us":126,"threads_started":1}
I20260925 14:53:04.655441 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b86394372148e8b808441f8175dccf" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3915efee0d08492c84356abdcdc1e4b1" is_pre_election: true
I20260925 14:53:04.655443 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b86394372148e8b808441f8175dccf" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6589127d13064be788c088a3f69fa096" is_pre_election: true
I20260925 14:53:04.655611 14553 raft_consensus.cc:2454] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 0.
I20260925 14:53:04.655622 14285 raft_consensus.cc:2454] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 0.
I20260925 14:53:04.655879 14354 leader_election.cc:302] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e; no voters:
I20260925 14:53:04.656038 14623 raft_consensus.cc:2781] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:04.656100 14623 raft_consensus.cc:482] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:04.656134 14623 raft_consensus.cc:3037] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:04.656821 14623 raft_consensus.cc:504] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.656985 14623 leader_election.cc:288] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 election: Requested vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:04.657101 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b86394372148e8b808441f8175dccf" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3915efee0d08492c84356abdcdc1e4b1"
I20260925 14:53:04.657178 14553 raft_consensus.cc:3037] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:04.657163 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4b86394372148e8b808441f8175dccf" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6589127d13064be788c088a3f69fa096"
I20260925 14:53:04.657261 14285 raft_consensus.cc:3037] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:04.657876 14553 raft_consensus.cc:2454] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 1.
I20260925 14:53:04.658038 14285 raft_consensus.cc:2454] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 1.
I20260925 14:53:04.658135 14354 leader_election.cc:302] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e; no voters:
I20260925 14:53:04.658216 14623 raft_consensus.cc:2781] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:04.658447 14623 raft_consensus.cc:686] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 1 LEADER]: Becoming Leader. State: Replica: 3987721521474cf2b9df1a6b691ba25e, State: Running, Role: LEADER
I20260925 14:53:04.658571 14623 consensus_queue.cc:237] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.659370 14140 catalog_manager.cc:5495] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3987721521474cf2b9df1a6b691ba25e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3987721521474cf2b9df1a6b691ba25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:04.670832 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.668432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.0.98.2:41617)
0925 14:53:04.668489 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.668490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.668498 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:04.668541 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.668542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.668558 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.668617 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:04.668740 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.669196 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.669905 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.670090 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.670142 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.670448 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:04.670604 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:04.670670 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.670712 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.670730 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:53:04.672824 14553 raft_consensus.cc:1261] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Refusing update from remote peer 3987721521474cf2b9df1a6b691ba25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:04.672824 14285 raft_consensus.cc:1261] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Refusing update from remote peer 3987721521474cf2b9df1a6b691ba25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:04.673166 14626 consensus_queue.cc:1035] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:04.673326 14623 consensus_queue.cc:1035] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:04.675781 14629 mvcc.cc:205] Tried to move back new op lower bound from 7333265345217007616 to 7333265345162006528. Current Snapshot: MvccSnapshot[applied={T|T < 7333265345217007616}]
I20260925 14:53:04.676127 14630 mvcc.cc:205] Tried to move back new op lower bound from 7333265345217007616 to 7333265345162006528. Current Snapshot: MvccSnapshot[applied={T|T < 7333265345217007616}]
I20260925 14:53:04.676563 14631 mvcc.cc:205] Tried to move back new op lower bound from 7333265345217007616 to 7333265345162006528. Current Snapshot: MvccSnapshot[applied={T|T < 7333265345217007616}]
I20260925 14:53:04.678071 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:04.678184 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47434:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:04.678985 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Sending DeleteTablet for 3 replicas of tablet d4b86394372148e8b808441f8175dccf
I20260925 14:53:04.679311 14533 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b86394372148e8b808441f8175dccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52418
I20260925 14:53:04.679359 14399 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b86394372148e8b808441f8175dccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43128
I20260925 14:53:04.679411 14265 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b86394372148e8b808441f8175dccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55042
I20260925 14:53:04.679589 14640 tablet_replica.cc:330] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: stopping tablet replica
I20260925 14:53:04.679618 14641 tablet_replica.cc:330] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: stopping tablet replica
I20260925 14:53:04.679661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14202
I20260925 14:53:04.679679 14640 raft_consensus.cc:2229] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:04.679711 14641 raft_consensus.cc:2229] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:04.679766 14640 raft_consensus.cc:2258] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:04.679811 14641 raft_consensus.cc:2258] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:04.681624 14640 ts_tablet_manager.cc:1905] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:04.682485 14639 tablet_replica.cc:330] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: stopping tablet replica
I20260925 14:53:04.682600 14639 raft_consensus.cc:2229] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:04.682708 14639 raft_consensus.cc:2258] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:04.683151 14639 ts_tablet_manager.cc:1905] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:04.683622 14640 ts_tablet_manager.cc:1918] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:04.683697 14640 log.cc:1196] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/d4b86394372148e8b808441f8175dccf
I20260925 14:53:04.683956 14640 ts_tablet_manager.cc:1939] T d4b86394372148e8b808441f8175dccf P 3987721521474cf2b9df1a6b691ba25e: Deleting consensus metadata
I20260925 14:53:04.684357 14126 catalog_manager.cc:4841] TS 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): tablet d4b86394372148e8b808441f8175dccf (table test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]) successfully deleted
I20260925 14:53:04.684938 14639 ts_tablet_manager.cc:1918] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:04.685037 14639 log.cc:1196] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/d4b86394372148e8b808441f8175dccf
I20260925 14:53:04.685331 14639 ts_tablet_manager.cc:1939] T d4b86394372148e8b808441f8175dccf P 3915efee0d08492c84356abdcdc1e4b1: Deleting consensus metadata
I20260925 14:53:04.685776 14126 catalog_manager.cc:4841] TS 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): tablet d4b86394372148e8b808441f8175dccf (table test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]) successfully deleted
W20260925 14:53:04.686071 14127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42137 (error 108)
I20260925 14:53:04.686651 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.686510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.0.1:55050)
0925 14:53:04.686574 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.686614 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:53:04.686664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14336
W20260925 14:53:04.688104 14127 catalog_manager.cc:4584] TS 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4b86394372148e8b808441f8175dccf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42137: connect: Connection refused (error 111)
I20260925 14:53:04.691922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14470
I20260925 14:53:04.696661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14108
I20260925 14:53:04.701815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--webserver_interface=127.0.98.62
--webserver_port=42997
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:04.783622 14643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.783824 14643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.783846 14643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.784924 14643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:04.784960 14643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.784973 14643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.784986 14643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:04.785003 14643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:04.786405 14643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:04.786592 14643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.786761 14643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.795831 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.795840 14652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.795835 14650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.796195 14643 server_base.cc:1034] running on GCE node
I20260925 14:53:04.796466 14643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.796783 14643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.797951 14643 hybrid_clock.cc:648] HybridClock initialized: now 1790347984797918 us; error 50 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.806113 14643 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:04.806686 14643 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260925 14:53:04.806941 14643 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.807036 14643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.808468 14643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:04.809204 14659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.809433 14643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:04.809554 14643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "384b9314247d48bdb56d0ec0682c75f8"
format_stamp: "Formatted at 2026-09-25 14:53:03 on dist-test-slave-gpc1"
server_key: "6620fd343d3e60e2e635cfe95d1f088c"
server_key_iv: "98a2233e2358fb864746edaa3f8b26ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.809976 14643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.827977 14643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.830011 14643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.830186 14643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.834429 14643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38245
I20260925 14:53:04.834437 14711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38245 every 8 connection(s)
I20260925 14:53:04.834908 14643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:04.836491 14712 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:53:04.836818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14643
I20260925 14:53:04.837154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42137
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=36601
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:04.837443 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap starting.
I20260925 14:53:04.845892 14712 log.cc:824] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.847988 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:04.848281 14712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap complete.
I20260925 14:53:04.850672 14712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:04.850934 14712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.851114 14712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:04.851191 14712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:04.851248 14712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:04.851300 14712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:53:04.852298 14712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:04.852417 14712 leader_election.cc:302] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384b9314247d48bdb56d0ec0682c75f8; no voters:
I20260925 14:53:04.852624 14712 leader_election.cc:288] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:53:04.852649 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:53:04.852764 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 LEADER]: Becoming Leader. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Running, Role: LEADER
I20260925 14:53:04.852849 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:04.852896 14712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:04.853107 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:04.853322 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:04.853264 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384b9314247d48bdb56d0ec0682c75f8. Latest consensus state: current_term: 2 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:04.853586 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:04.853674 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:04.854485 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:04.854583 14724 catalog_manager.cc:1259] Loaded cluster ID: 4928e8ae22f44470a41fb4a8a019c5f4
I20260925 14:53:04.854607 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:04.854830 14724 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 14:53:04.859474 14724 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 14:53:04.861814 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:04.862056 14724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Loaded TSK: 0
I20260925 14:53:04.862287 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:53:04.921185 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.921433 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.921475 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.922747 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.922809 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.922856 14714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:53:04.924371 14714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36601
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:04.924634 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.924865 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.930716 14740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.930751 14742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.930751 14739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.931025 14714 server_base.cc:1034] running on GCE node
I20260925 14:53:04.931241 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.931514 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.932698 14714 hybrid_clock.cc:648] HybridClock initialized: now 1790347984932678 us; error 32 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.935626 14714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:53:04.936097 14714 webserver.cc:460] Webserver started at http://127.0.98.1:36601/ using document root <none> and password file <none>
I20260925 14:53:04.936326 14714 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.936408 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.937806 14714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:04.938580 14749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.938802 14714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:04.938916 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "6589127d13064be788c088a3f69fa096"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "bfe47555622a92e4e72a57c88a822e7c"
server_key_iv: "dd3b6bed5031ab8f91f03ff13bb26ee2"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.939283 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.957504 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.960135 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.960318 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.960577 14714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:04.961054 14756 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:53:04.962985 14714 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:53:04.963063 14714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:53:04.963109 14714 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:53:04.963783 14714 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:53:04.963840 14714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:04.963917 14756 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Bootstrap starting.
I20260925 14:53:04.971319 14714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42137
I20260925 14:53:04.971697 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:04.972360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14714
I20260925 14:53:04.972673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41617
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=45167
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:04.974529 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42137 every 8 connection(s)
I20260925 14:53:04.974565 14756 log.cc:824] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.977186 14756 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:04.977466 14756 tablet_bootstrap.cc:492] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Bootstrap complete.
I20260925 14:53:04.978669 14756 ts_tablet_manager.cc:1397] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260925 14:53:04.980023 14756 raft_consensus.cc:348] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.980337 14756 raft_consensus.cc:729] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6589127d13064be788c088a3f69fa096, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.980485 14756 consensus_queue.cc:260] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:04.980733 14756 ts_tablet_manager.cc:1428] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:04.987929 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.972438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:34841)
0925 14:53:04.975173 (+ 2735us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.975204 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.975713 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.976275 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.976279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.976690 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:04.976970 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.976977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.978833 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.978834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.980271 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.980274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.981156 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.981157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.981326 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.981982 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:04.981992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:04.983957 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:04.986118 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.986125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.986127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.986471 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.986746 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.986746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.986747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.986872 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.987298 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:04.987303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:04.987453 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 14:53:04.987559 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:53:04.987728 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2697,"thread_start_us":95,"threads_started":1}
I20260925 14:53:04.988698 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.972504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34841 (local address 127.0.98.62:38245)
0925 14:53:04.974894 (+ 2390us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.974905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.975336 (+ 431us) server_negotiation.cc:407] Connection header received
0925 14:53:04.975939 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.975946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.976006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.976136 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.977106 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.978717 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.980373 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.981039 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.981156 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.984070 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.984074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.984074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.984085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.986029 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.986556 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.986559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.986559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.986666 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.986946 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.986947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.986947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.987229 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.987303 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.987640 (+ 337us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.987794 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2336,"thread_start_us":78,"threads_started":1}
I20260925 14:53:04.989552 14864 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:04.989638 14864 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:04.989840 14864 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
I20260925 14:53:04.990634 14676 ts_manager.cc:194] Registered new tserver with Master: 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:04.991710 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34841
I20260925 14:53:04.992596 14864 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
Sep 25 14:53:04 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:53:05.000174 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.991224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.0.1:55064)
0925 14:53:04.991686 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:04.991720 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:04.991795 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:04.992210 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:04.992215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:04.992563 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:04.992618 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.992620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.993905 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.993906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.994605 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:04.994608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.995209 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.995211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:04.995268 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.995761 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:04.995765 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:04.997233 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:04.999123 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.999129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.999131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.999325 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.999542 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:04.999543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:04.999543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:04.999595 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:04.999898 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:04.999902 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:04.999946 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.000006 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.000098 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":93,"threads_started":1}
I20260925 14:53:05.000252 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.991449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.0.98.1:42137)
0925 14:53:04.991621 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.991625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.991867 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:53:04.991902 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.991904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.991917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.992154 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.992704 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.993809 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.994678 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:04.995125 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:04.995200 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:04.997323 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.997331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.997331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:04.997339 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.999043 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.999405 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.999408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.999408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.999473 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:04.999661 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:04.999662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:04.999662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:04.999835 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.999916 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.000061 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.000159 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260925 14:53:05.000643 14798 tablet_service.cc:1514] Processing DeleteTablet for tablet d4b86394372148e8b808441f8175dccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260925 14:53:05.001235 14873 tablet_replica.cc:330] stopping tablet replica
I20260925 14:53:05.001354 14873 raft_consensus.cc:2229] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:05.001435 14873 raft_consensus.cc:2258] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:05.001742 14873 ts_tablet_manager.cc:1905] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:05.003788 14873 ts_tablet_manager.cc:1918] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:05.003877 14873 log.cc:1196] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/d4b86394372148e8b808441f8175dccf
I20260925 14:53:05.004078 14873 ts_tablet_manager.cc:1939] T d4b86394372148e8b808441f8175dccf P 6589127d13064be788c088a3f69fa096: Deleting consensus metadata
I20260925 14:53:05.004588 14663 catalog_manager.cc:4841] TS 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137): tablet d4b86394372148e8b808441f8175dccf (table test-table [id=ffcb3d22f6de435ab65ac8b67348ecff]) successfully deleted
W20260925 14:53:05.061244 14868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.061455 14868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.061477 14868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.062580 14868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.062614 14868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.062640 14868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:53:05.064021 14868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45167
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.064208 14868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.064378 14868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.069391 14881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.069558 14868 server_base.cc:1034] running on GCE node
W20260925 14:53:05.069407 14879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.069413 14878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.069846 14868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.070148 14868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.071313 14868 hybrid_clock.cc:648] HybridClock initialized: now 1790347985071294 us; error 35 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.073819 14868 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:53:05.074318 14868 webserver.cc:460] Webserver started at http://127.0.98.2:45167/ using document root <none> and password file <none>
I20260925 14:53:05.074582 14868 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.074664 14868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.076061 14868 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.076774 14888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.076969 14868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.077106 14868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "3987721521474cf2b9df1a6b691ba25e"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "8ec16c349fb99fc6357455e677043818"
server_key_iv: "d865ff706448ce252d9bafc8594db3b9"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.077453 14868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.086716 14868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.088573 14868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.088732 14868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.088960 14868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:05.089264 14868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:05.089318 14868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.089359 14868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:05.089387 14868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.095558 14868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41617
I20260925 14:53:05.095597 15001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41617 every 8 connection(s)
I20260925 14:53:05.095953 14868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:05.099164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 14868
I20260925 14:53:05.099472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35043
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=42557
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:05.106454 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.096708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33265 (local address 127.0.98.62:38245)
0925 14:53:05.096774 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.096776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.096996 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:53:05.097500 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.097501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.097513 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.097577 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:05.098508 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.099015 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.100533 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.100794 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.100873 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.103184 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:05.103186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:05.103186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:05.103195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:05.104939 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.105348 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.105349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.105350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.105418 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.105676 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.105677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.105677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.105839 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:05.105919 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.106192 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.106294 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:53:05.106489 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.096693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:33265)
0925 14:53:05.096848 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.096884 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.097411 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.097632 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.097635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.098198 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:05.098430 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.098435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.099105 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.099106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.100433 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.100436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.100893 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.100894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.101036 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.101619 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:05.101628 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:05.103088 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:05.105025 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:05.105052 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:05.105054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:05.105270 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:05.105509 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:05.105510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:05.105510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:05.105609 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:05.105910 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:05.105914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:05.106027 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.106142 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.106264 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260925 14:53:05.107120 15002 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:05.107234 15002 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:05.107403 15002 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
I20260925 14:53:05.107792 14676 ts_manager.cc:194] Registered new tserver with Master: 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
I20260925 14:53:05.108188 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33265
W20260925 14:53:05.183341 15005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.183581 15005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.183621 15005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.184856 15005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.184914 15005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.184959 15005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:05.186478 15005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42557
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.186749 15005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.186998 15005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.191948 15011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.192005 15013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.192287 15005 server_base.cc:1034] running on GCE node
W20260925 14:53:05.191967 15010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.192553 15005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.192761 15005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.193933 15005 hybrid_clock.cc:648] HybridClock initialized: now 1790347985193922 us; error 40 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.196434 15005 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:53:05.196918 15005 webserver.cc:460] Webserver started at http://127.0.98.3:42557/ using document root <none> and password file <none>
I20260925 14:53:05.197162 15005 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.197253 15005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.198630 15005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.199206 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.199378 15005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:05.199478 15005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "3915efee0d08492c84356abdcdc1e4b1"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "7e87b9ef4e5bb3a60f12f61bf3cdd2fa"
server_key_iv: "60fba0c1ecebbdba48084aaf1622264e"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.199837 15005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.211439 15005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.213371 15005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.213541 15005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.213788 15005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:05.214099 15005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:05.214152 15005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.214196 15005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:05.214221 15005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.220098 15005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35043
I20260925 14:53:05.220145 15133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35043 every 8 connection(s)
I20260925 14:53:05.220484 15005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:05.225246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15005
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:05.230433 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.221181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57597 (local address 127.0.98.62:38245)
0925 14:53:05.221255 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.221257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.221515 (+ 258us) server_negotiation.cc:407] Connection header received
0925 14:53:05.221982 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.221983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.221993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.222093 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:05.223122 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.223601 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.224580 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.224775 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.224865 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.227518 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:05.227519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:05.227520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:05.227535 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:05.229017 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.229425 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.229427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.229427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.229487 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.229750 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.229751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.229752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.229859 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:05.229933 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.230219 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.230305 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:05.230621 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.221187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:57597)
0925 14:53:05.221384 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.221422 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.221899 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.222170 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.222173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.222827 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:05.223036 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.223041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.223682 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.223682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.224490 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.224492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.224876 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.224877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.224986 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.225595 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:05.225606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:05.227432 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:05.229097 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:05.229103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:05.229105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:05.229346 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:05.229571 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:05.229572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:05.229572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:05.229681 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:05.229925 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:05.229931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:05.230084 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.230195 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.230379 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260925 14:53:05.231108 15134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38245
I20260925 14:53:05.231220 15134 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:05.231418 15134 heartbeater.cc:511] Master 127.0.98.62:38245 requested a full tablet report, sending...
I20260925 14:53:05.231817 14676 ts_manager.cc:194] Registered new tserver with Master: 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043)
I20260925 14:53:05.232254 14676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57597
I20260925 14:53:05.233296 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.225686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.0.98.62:38245)
0925 14:53:05.225801 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.225803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.225838 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:05.225918 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.225919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.225928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.225979 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:05.226693 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.227421 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.228522 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.228722 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.228783 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.230018 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:05.230020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:05.230020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:05.230033 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:05.232008 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.232319 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.232320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.232321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.232380 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.232715 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.232716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.232716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.232842 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:05.233063 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.233108 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.233201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":48,"threads_started":1}
I20260925 14:53:05.233956 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:05.236488 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.234387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.0.98.62:38245)
0925 14:53:05.234424 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.234425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.234553 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:53:05.234587 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.234588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.234613 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.234669 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:05.234753 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.235187 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.235848 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.236058 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.236107 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.236221 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:05.236303 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:05.236355 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.236383 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.236398 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:53:05.236771 14676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:05.238409 14676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:05.240443 14798 tablet_service.cc:1467] Processing CreateTablet for tablet da89f6f3b2324839847d0320d3ce57ec (DEFAULT_TABLE table=test-table [id=541124597f9d40e78abded3e13536523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:05.240661 14798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89f6f3b2324839847d0320d3ce57ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:05.242050 14756 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Bootstrap starting.
I20260925 14:53:05.242380 14756 tablet_bootstrap.cc:654] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:05.243237 14756 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: No bootstrap required, opened a new log
I20260925 14:53:05.243307 14756 ts_tablet_manager.cc:1397] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:05.243464 14756 raft_consensus.cc:348] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.243551 14756 raft_consensus.cc:374] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:05.243603 14756 raft_consensus.cc:729] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6589127d13064be788c088a3f69fa096, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.243647 14756 consensus_queue.cc:260] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.243742 14756 ts_tablet_manager.cc:1428] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:05.244875 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.240592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.0.98.2:41617)
0925 14:53:05.240765 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.240771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.240780 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:05.240819 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.240821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.240836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.241056 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:05.241124 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.242237 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.243655 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.244412 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.244497 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.244559 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.244751 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.244792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260925 14:53:05.244875 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.240318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.98.3:35043)
0925 14:53:05.240643 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.240662 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.240670 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:05.240728 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.240730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.240746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.240864 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:05.240996 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.242077 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.243632 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.244440 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.244505 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.244559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.244751 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.244792 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":92,"threads_started":1}
I20260925 14:53:05.244870 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.240320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41617 (local address 127.0.0.1:43134)
0925 14:53:05.240474 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.240485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.240531 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.240934 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.240935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.240937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:05.240982 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.240984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.242388 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.242389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.243538 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.243540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.244556 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.244556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.244632 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.244655 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.244722 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.244772 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":9,"thread_start_us":114,"threads_started":1}
I20260925 14:53:05.244869 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.240224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35043 (local address 127.0.0.1:52420)
0925 14:53:05.240255 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.240266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.240299 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.240876 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.240877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.240880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:05.240917 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.240919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.242281 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.242282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.243537 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.243540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.244555 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.244556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.244636 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.244655 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.244722 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.244772 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":33}
I20260925 14:53:05.245589 14936 tablet_service.cc:1467] Processing CreateTablet for tablet da89f6f3b2324839847d0320d3ce57ec (DEFAULT_TABLE table=test-table [id=541124597f9d40e78abded3e13536523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:05.245592 15068 tablet_service.cc:1467] Processing CreateTablet for tablet da89f6f3b2324839847d0320d3ce57ec (DEFAULT_TABLE table=test-table [id=541124597f9d40e78abded3e13536523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:05.245932 15068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89f6f3b2324839847d0320d3ce57ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:05.245932 14936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da89f6f3b2324839847d0320d3ce57ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:05.249516 15145 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Bootstrap starting.
I20260925 14:53:05.250113 15145 tablet_bootstrap.cc:654] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:05.250525 15144 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Bootstrap starting.
I20260925 14:53:05.250761 15145 log.cc:824] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:05.251222 15144 tablet_bootstrap.cc:654] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:05.251591 15145 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: No bootstrap required, opened a new log
I20260925 14:53:05.251675 15145 ts_tablet_manager.cc:1397] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:05.251811 15144 log.cc:824] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:05.252629 15144 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: No bootstrap required, opened a new log
I20260925 14:53:05.252697 15144 ts_tablet_manager.cc:1397] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:05.253842 15145 raft_consensus.cc:348] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.254061 15145 raft_consensus.cc:374] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:05.254104 15145 raft_consensus.cc:729] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3987721521474cf2b9df1a6b691ba25e, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.254230 15145 consensus_queue.cc:260] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.254549 15145 ts_tablet_manager.cc:1428] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:53:05.254572 15002 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
I20260925 14:53:05.254794 15144 raft_consensus.cc:348] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.254963 15144 raft_consensus.cc:374] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:05.255018 15144 raft_consensus.cc:729] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3915efee0d08492c84356abdcdc1e4b1, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.255131 15144 consensus_queue.cc:260] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.255410 15144 ts_tablet_manager.cc:1428] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:05.255484 15134 heartbeater.cc:503] Master 127.0.98.62:38245 was elected leader, sending a full tablet report...
I20260925 14:53:05.256466 15148 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:05.256572 15148 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.256913 15148 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:05.260830 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.257247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45045 (local address 127.0.98.3:35043)
0925 14:53:05.257293 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.257295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.257303 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:05.257327 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.257328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.257340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.257391 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:05.257609 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.258415 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.259902 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.260466 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.260546 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.260560 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.260725 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.260761 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:05.260833 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.257164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53699 (local address 127.0.98.1:42137)
0925 14:53:05.257211 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.257213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.257348 (+ 135us) server_negotiation.cc:407] Connection header received
0925 14:53:05.257459 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.257461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.257490 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.257549 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:05.257701 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.258438 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.259903 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.260510 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.260582 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.260653 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.260725 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.260762 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:05.260857 15004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.257033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35043 (local address 127.0.98.2:45045)
0925 14:53:05.257081 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.257092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.257132 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.257440 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.257441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.257443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:05.257508 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.257510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.258496 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.258497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.259808 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.259811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.260547 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.260548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.260648 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.260652 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.260712 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.260760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":91}
I20260925 14:53:05.260866 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.257109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.98.2:53699)
0925 14:53:05.257281 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.257292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.257372 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.257563 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:05.257564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:05.257566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:05.257626 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.257627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.258505 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.258505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.259808 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:05.259811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.260587 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.260588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:05.260668 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.260671 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:05.260723 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:53:05.260760 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":42,"thread_start_us":85,"threads_started":1}
I20260925 14:53:05.261251 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89f6f3b2324839847d0320d3ce57ec" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3915efee0d08492c84356abdcdc1e4b1" is_pre_election: true
I20260925 14:53:05.261396 15088 raft_consensus.cc:2454] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 0.
I20260925 14:53:05.261387 14818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89f6f3b2324839847d0320d3ce57ec" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6589127d13064be788c088a3f69fa096" is_pre_election: true
I20260925 14:53:05.261559 14818 raft_consensus.cc:2454] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 0.
I20260925 14:53:05.261629 14891 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e; no voters:
I20260925 14:53:05.261778 15148 raft_consensus.cc:2781] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:05.261843 15148 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:05.261870 15148 raft_consensus.cc:3037] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:05.262748 15148 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.262888 15148 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 election: Requested vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137)
I20260925 14:53:05.263010 14818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89f6f3b2324839847d0320d3ce57ec" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6589127d13064be788c088a3f69fa096"
I20260925 14:53:05.263104 14818 raft_consensus.cc:3037] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:05.263043 15088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da89f6f3b2324839847d0320d3ce57ec" candidate_uuid: "3987721521474cf2b9df1a6b691ba25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3915efee0d08492c84356abdcdc1e4b1"
I20260925 14:53:05.263176 15088 raft_consensus.cc:3037] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:05.263821 14818 raft_consensus.cc:2454] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 1.
I20260925 14:53:05.263890 15088 raft_consensus.cc:2454] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3987721521474cf2b9df1a6b691ba25e in term 1.
I20260925 14:53:05.263985 14892 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3987721521474cf2b9df1a6b691ba25e, 6589127d13064be788c088a3f69fa096; no voters:
I20260925 14:53:05.264081 15148 raft_consensus.cc:2781] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:05.264220 15148 raft_consensus.cc:686] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 1 LEADER]: Becoming Leader. State: Replica: 3987721521474cf2b9df1a6b691ba25e, State: Running, Role: LEADER
I20260925 14:53:05.264322 15148 consensus_queue.cc:237] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.265097 14676 catalog_manager.cc:5495] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3987721521474cf2b9df1a6b691ba25e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3987721521474cf2b9df1a6b691ba25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:05.273072 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.270743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.0.98.2:41617)
0925 14:53:05.270797 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.270799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.270826 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:53:05.270920 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.270921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.270940 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.271028 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:05.271130 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.271592 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.272314 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.272512 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.272609 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.272760 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:05.272855 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:05.272946 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.272966 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.272986 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:53:05.274688 15088 raft_consensus.cc:1261] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Refusing update from remote peer 3987721521474cf2b9df1a6b691ba25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:05.274695 14818 raft_consensus.cc:1261] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Refusing update from remote peer 3987721521474cf2b9df1a6b691ba25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:05.275033 15148 consensus_queue.cc:1035] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:05.275172 15151 consensus_queue.cc:1035] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:05.277557 15155 mvcc.cc:205] Tried to move back new op lower bound from 7333265347682742272 to 7333265347643281408. Current Snapshot: MvccSnapshot[applied={T|T < 7333265347682742272}]
I20260925 14:53:05.278244 15154 mvcc.cc:205] Tried to move back new op lower bound from 7333265347682742272 to 7333265347643281408. Current Snapshot: MvccSnapshot[applied={T|T < 7333265347682742272}]
I20260925 14:53:05.280234 14676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:05.280325 14676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:05.281008 14676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Sending DeleteTablet for 3 replicas of tablet da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:05.281329 14798 tablet_service.cc:1514] Processing DeleteTablet for tablet da89f6f3b2324839847d0320d3ce57ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55064
I20260925 14:53:05.281356 15068 tablet_service.cc:1514] Processing DeleteTablet for tablet da89f6f3b2324839847d0320d3ce57ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52420
I20260925 14:53:05.281425 14873 tablet_replica.cc:330] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: stopping tablet replica
I20260925 14:53:05.281479 14873 raft_consensus.cc:2229] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:05.281553 14873 raft_consensus.cc:2258] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:05.281618 14936 tablet_service.cc:1514] Processing DeleteTablet for tablet da89f6f3b2324839847d0320d3ce57ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43134
I20260925 14:53:05.281675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14714
I20260925 14:53:05.281898 15165 tablet_replica.cc:330] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: stopping tablet replica
I20260925 14:53:05.282018 15165 raft_consensus.cc:2229] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:05.282191 15165 raft_consensus.cc:2258] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:05.282697 15165 ts_tablet_manager.cc:1905] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:05.283246 15164 tablet_replica.cc:330] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: stopping tablet replica
I20260925 14:53:05.283351 15164 raft_consensus.cc:2229] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:05.283456 15164 raft_consensus.cc:2258] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:05.283883 15164 ts_tablet_manager.cc:1905] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:05.284617 15165 ts_tablet_manager.cc:1918] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:05.284734 15165 log.cc:1196] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:05.285080 15165 ts_tablet_manager.cc:1939] T da89f6f3b2324839847d0320d3ce57ec P 3987721521474cf2b9df1a6b691ba25e: Deleting consensus metadata
I20260925 14:53:05.285529 14662 catalog_manager.cc:4841] TS 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): tablet da89f6f3b2324839847d0320d3ce57ec (table test-table [id=541124597f9d40e78abded3e13536523]) successfully deleted
I20260925 14:53:05.285887 15164 ts_tablet_manager.cc:1918] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:05.285960 15164 log.cc:1196] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:05.286259 15164 ts_tablet_manager.cc:1939] T da89f6f3b2324839847d0320d3ce57ec P 3915efee0d08492c84356abdcdc1e4b1: Deleting consensus metadata
I20260925 14:53:05.286684 14662 catalog_manager.cc:4841] TS 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): tablet da89f6f3b2324839847d0320d3ce57ec (table test-table [id=541124597f9d40e78abded3e13536523]) successfully deleted
W20260925 14:53:05.288127 14663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42137 (error 108)
I20260925 14:53:05.288512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14868
I20260925 14:53:05.288637 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.288488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42137 (local address 127.0.0.1:55068)
0925 14:53:05.288559 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.288601 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 14:53:05.289445 14663 catalog_manager.cc:4584] TS 6589127d13064be788c088a3f69fa096 (127.0.98.1:42137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da89f6f3b2324839847d0320d3ce57ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42137: connect: Connection refused (error 111)
I20260925 14:53:05.293807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15005
I20260925 14:53:05.298188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 14643
I20260925 14:53:05.303437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--webserver_interface=127.0.98.62
--webserver_port=42997
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:05.384359 15167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.384593 15167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.384631 15167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.385752 15167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:05.385818 15167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.385843 15167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.385874 15167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:05.385898 15167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:05.387415 15167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.387686 15167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.387923 15167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.392781 15172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.392817 15173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.392783 15175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.393091 15167 server_base.cc:1034] running on GCE node
I20260925 14:53:05.393332 15167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.393610 15167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.394767 15167 hybrid_clock.cc:648] HybridClock initialized: now 1790347985394759 us; error 37 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.397137 15167 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:05.397606 15167 webserver.cc:460] Webserver started at http://127.0.98.62:42997/ using document root <none> and password file <none>
I20260925 14:53:05.397825 15167 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.397892 15167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.399319 15167 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.400040 15182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.400338 15167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.400470 15167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "384b9314247d48bdb56d0ec0682c75f8"
format_stamp: "Formatted at 2026-09-25 14:53:03 on dist-test-slave-gpc1"
server_key: "6620fd343d3e60e2e635cfe95d1f088c"
server_key_iv: "98a2233e2358fb864746edaa3f8b26ab"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.400856 15167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.413650 15167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.415592 15167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.415757 15167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.420158 15167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38245
I20260925 14:53:05.420192 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38245 every 8 connection(s)
I20260925 14:53:05.420583 15167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:05.422118 15235 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:53:05.422861 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap starting.
I20260925 14:53:05.428022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15167
I20260925 14:53:05.428356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42137
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=36601
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:05.431628 15235 log.cc:824] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:05.433842 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:05.434168 15235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8: Bootstrap complete.
I20260925 14:53:05.436762 15235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:05.437018 15235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.437198 15235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:05.437278 15235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:05.437347 15235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:05.437400 15235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:53:05.438519 15235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:05.438620 15235 leader_election.cc:302] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384b9314247d48bdb56d0ec0682c75f8; no voters:
I20260925 14:53:05.438861 15235 leader_election.cc:288] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:53:05.438901 15239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:53:05.439050 15239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [term 3 LEADER]: Becoming Leader. State: Replica: 384b9314247d48bdb56d0ec0682c75f8, State: Running, Role: LEADER
I20260925 14:53:05.439150 15235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:05.439148 15239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } }
I20260925 14:53:05.439738 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:05.439844 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:05.440058 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384b9314247d48bdb56d0ec0682c75f8. Latest consensus state: current_term: 3 leader_uuid: "384b9314247d48bdb56d0ec0682c75f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384b9314247d48bdb56d0ec0682c75f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38245 } } }
I20260925 14:53:05.440346 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384b9314247d48bdb56d0ec0682c75f8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:05.440050 15247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:05.441179 15247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:05.441321 15247 catalog_manager.cc:1259] Loaded cluster ID: 4928e8ae22f44470a41fb4a8a019c5f4
I20260925 14:53:05.441588 15247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 14:53:05.441946 15247 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f72f4c219ad google::LogMessage::Fail() at ??:0
@ 0x7f72f4c2488a google::LogMessage::SendToLog() at ??:0
@ 0x7f72f4c214de google::LogMessage::Flush() at ??:0
@ 0x7f72f4c22a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f72fcb420f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f72fcb4b8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f72f6aef0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f72f6ae4954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f72f79486db start_thread at ??:0
@ 0x7f72f371f61f clone at ??:0
W20260925 14:53:05.525843 15237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.526049 15237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.526074 15237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.527304 15237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.527347 15237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.527374 15237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:53:05.528829 15237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36601
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.529068 15237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.529322 15237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.534277 15274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.534276 15271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.534274 15272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.534776 15237 server_base.cc:1034] running on GCE node
I20260925 14:53:05.534991 15237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.535274 15237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.536459 15237 hybrid_clock.cc:648] HybridClock initialized: now 1790347985536444 us; error 29 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.539628 15237 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:53:05.540134 15237 webserver.cc:460] Webserver started at http://127.0.98.1:36601/ using document root <none> and password file <none>
I20260925 14:53:05.540373 15237 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.540431 15237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.542178 15237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:05.542975 15281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.543169 15237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:05.543251 15237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "6589127d13064be788c088a3f69fa096"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "bfe47555622a92e4e72a57c88a822e7c"
server_key_iv: "dd3b6bed5031ab8f91f03ff13bb26ee2"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.543602 15237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.551564 15237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.553699 15237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.553862 15237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.554100 15237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:05.554626 15288 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:53:05.556643 15237 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:53:05.556707 15237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:53:05.556751 15237 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:53:05.557394 15237 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:53:05.557446 15237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:05.557518 15288 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Bootstrap starting.
I20260925 14:53:05.564491 15237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42137
I20260925 14:53:05.564634 15395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42137 every 8 connection(s)
I20260925 14:53:05.564900 15237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:05.568907 15288 log.cc:824] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:05.571710 15288 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:05.572019 15288 tablet_bootstrap.cc:492] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Bootstrap complete.
I20260925 14:53:05.573240 15288 ts_tablet_manager.cc:1397] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.011s
I20260925 14:53:05.574081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15237
I20260925 14:53:05.574417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41617
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=45167
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:05.574843 15288 raft_consensus.cc:348] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.575176 15288 raft_consensus.cc:729] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6589127d13064be788c088a3f69fa096, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.575317 15288 consensus_queue.cc:260] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:05.575666 15288 ts_tablet_manager.cc:1428] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
W20260925 14:53:05.649874 15398 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:05.565626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:52385)
0925 14:53:05.565820 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.565858 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:05.566291 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:05.649668 (+ 83377us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
W20260925 14:53:05.650166 15396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38245: Client connection negotiation failed: client connection to 127.0.98.62:38245: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:53:05.650575 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.650454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:56069)
0925 14:53:05.650506 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.650554 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:53:05.650925 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.650845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:42703)
0925 14:53:05.650888 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.650911 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:53:05.651080 15396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:53:05.657327 15401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.657516 15401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.657539 15401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.658685 15401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.658725 15401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.658751 15401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:53:05.660130 15401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45167
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.660320 15401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.660497 15401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.665416 15406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.665416 15409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.665454 15407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.665637 15401 server_base.cc:1034] running on GCE node
I20260925 14:53:05.665904 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.666154 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.667342 15401 hybrid_clock.cc:648] HybridClock initialized: now 1790347985667329 us; error 31 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.669977 15401 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:53:05.670519 15401 webserver.cc:460] Webserver started at http://127.0.98.2:45167/ using document root <none> and password file <none>
I20260925 14:53:05.670774 15401 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.670842 15401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.672196 15401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:05.672803 15416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.672981 15401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:05.673081 15401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "3987721521474cf2b9df1a6b691ba25e"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "8ec16c349fb99fc6357455e677043818"
server_key_iv: "d865ff706448ce252d9bafc8594db3b9"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.673444 15401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.702991 15401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.704957 15401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.705122 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.705359 15401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:05.705667 15401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:05.705718 15401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.705760 15401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:05.705787 15401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.711705 15401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41617
I20260925 14:53:05.711756 15529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41617 every 8 connection(s)
I20260925 14:53:05.712081 15401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:05.713186 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.712746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:33077)
0925 14:53:05.712902 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.713001 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
W20260925 14:53:05.713575 15530 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38245: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
I20260925 14:53:05.713841 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.713748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:48363)
0925 14:53:05.713794 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.713819 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:53:05.714108 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.714025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:47151)
0925 14:53:05.714068 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.714090 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:53:05.714212 15530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:53:05.720116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15401
I20260925 14:53:05.720444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35043
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38245
--webserver_port=42557
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:05.808327 15533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.808529 15533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.808553 15533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.809628 15533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.809660 15533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.809685 15533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:05.811092 15533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42557
--tserver_master_addrs=127.0.98.62:38245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:05.811277 15533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.811456 15533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.816568 15538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.816553 15539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.816548 15541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.816823 15533 server_base.cc:1034] running on GCE node
I20260925 14:53:05.817015 15533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.817268 15533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.818423 15533 hybrid_clock.cc:648] HybridClock initialized: now 1790347985818409 us; error 28 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-gpc1 krb5kdc[14095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.820943 15533 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:53:05.821388 15533 webserver.cc:460] Webserver started at http://127.0.98.3:42557/ using document root <none> and password file <none>
I20260925 14:53:05.821617 15533 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.821689 15533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.823135 15533 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:05.823792 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.824046 15533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:05.824150 15533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "3915efee0d08492c84356abdcdc1e4b1"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-gpc1"
server_key: "7e87b9ef4e5bb3a60f12f61bf3cdd2fa"
server_key_iv: "60fba0c1ecebbdba48084aaf1622264e"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.824515 15533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.862133 15533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.864486 15533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.864692 15533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.864967 15533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:05.865327 15533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:05.865386 15533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.865430 15533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:05.865499 15533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.871867 15533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35043
I20260925 14:53:05.871953 15661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35043 every 8 connection(s)
I20260925 14:53:05.872347 15533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:05.873493 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.873033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:60931)
0925 14:53:05.873253 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.873345 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
W20260925 14:53:05.873927 15662 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38245 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38245: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
I20260925 14:53:05.874226 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.874140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:57867)
0925 14:53:05.874182 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.874209 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:53:05.874558 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.874446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:56921)
0925 14:53:05.874507 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:05.874539 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 14:53:05.874709 15662 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:53:05.876201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15533
I20260925 14:53:06.651695 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:06.651455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:53915)
0925 14:53:06.651593 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:06.651634 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260925 14:53:06.714820 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:06.714587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:40429)
0925 14:53:06.714731 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:06.714769 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260925 14:53:06.875265 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:06.875043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:36477)
0925 14:53:06.875173 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:06.875212 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260925 14:53:07.443459 15669 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:07.443655 15669 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:07.444000 15669 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
Sep 25 14:53:07 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 25 14:53:07 dist-test-slave-gpc1 krb5kdc[14095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260925 14:53:07.455160 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.444562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58227 (local address 127.0.98.2:41617)
0925 14:53:07.444699 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.444707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.444723 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:53:07.445114 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.445115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.445186 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.445322 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.445976 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.446937 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448159 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.448550 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448659 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.451852 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.451856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.451856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:07.451866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.453471 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.453929 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.453933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.453933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.454022 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.454256 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.454257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.454257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.454549 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.454639 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.454921 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.455044 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":56,"threads_started":1}
I20260925 14:53:07.455155 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.444402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41617 (local address 127.0.98.1:58227)
0925 14:53:07.444486 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.444500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:07.444537 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:07.445312 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:07.445313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:07.445703 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:07.445859 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.445865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.447024 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.447025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.448073 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.448075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.448635 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.448746 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.449390 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:07.449411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:07.451714 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:07.453594 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.453595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.453596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.453835 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.454092 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.454093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.454093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.454186 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.454628 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:07.454634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:07.454762 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 14:53:07.454871 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:53:07.455060 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":24,"thread_start_us":51,"threads_started":1}
I20260925 14:53:07.455196 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.444319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53487 (local address 127.0.98.3:35043)
0925 14:53:07.444503 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.444515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.444541 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:53:07.445001 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.445004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.445045 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.445162 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.445984 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.447102 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448166 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.448558 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448675 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.451316 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.451320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.451320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:07.451330 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.453397 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.453890 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.453893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.453893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.453991 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:07.454254 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:07.454255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:07.454255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:07.454549 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.454640 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.454924 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.455075 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260925 14:53:07.455271 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.444210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35043 (local address 127.0.98.1:53487)
0925 14:53:07.444325 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.444341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:07.444397 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:07.445240 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:07.445244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:07.445683 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:07.445859 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.445865 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.447198 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.447198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.448072 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:07.448075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.448654 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.448655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:07.448752 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.449389 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:07.449398 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:07.451202 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:07.453513 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.453520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.453522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.453806 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.454065 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:07.454065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:07.454066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:07.454184 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:07.454623 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:07.454630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:07.454762 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 14:53:07.454871 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:53:07.455140 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":132,"thread_start_us":81,"threads_started":1}
W20260925 14:53:07.456539 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
W20260925 14:53:07.456611 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:07.456629 15284 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6589127d13064be788c088a3f69fa096; no voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e
I20260925 14:53:07.456717 15669 raft_consensus.cc:2726] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:53:07.652326 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.652169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:58569)
0925 14:53:07.652245 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.652284 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:53:07.715598 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.715351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:32841)
0925 14:53:07.715505 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.715553 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260925 14:53:07.875967 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.875734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:53013)
0925 14:53:07.875871 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:07.875920 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260925 14:53:08.653056 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.652834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:36763)
0925 14:53:08.652954 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:08.652988 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260925 14:53:08.716280 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.716090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:43093)
0925 14:53:08.716208 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:08.716241 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260925 14:53:08.876746 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.876447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:33909)
0925 14:53:08.876657 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:08.876702 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260925 14:53:09.272589 15679 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:09.272730 15679 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:09.272926 15679 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
W20260925 14:53:09.273439 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
W20260925 14:53:09.273504 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:09.273554 15284 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6589127d13064be788c088a3f69fa096; no voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e
I20260925 14:53:09.273645 15679 raft_consensus.cc:2726] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:53:09.653738 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:09.653503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:38043)
0925 14:53:09.653643 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:09.653676 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260925 14:53:09.717021 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:09.716760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:50429)
0925 14:53:09.716892 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:09.716929 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260925 14:53:09.877460 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:09.877224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:45467)
0925 14:53:09.877348 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:09.877391 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260925 14:53:10.654654 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.654437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:52741)
0925 14:53:10.654568 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.654607 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260925 14:53:10.717687 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.717464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:51541)
0925 14:53:10.717606 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.717647 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260925 14:53:10.878149 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:10.877912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:57129)
0925 14:53:10.878067 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:10.878105 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260925 14:53:11.369275 15686 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:11.369369 15686 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:11.369552 15686 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
W20260925 14:53:11.369937 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
W20260925 14:53:11.370049 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:11.370085 15284 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6589127d13064be788c088a3f69fa096; no voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e
I20260925 14:53:11.370148 15686 raft_consensus.cc:2726] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:53:11.655405 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.655157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:44903)
0925 14:53:11.655310 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.655358 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260925 14:53:11.718436 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.718163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:46197)
0925 14:53:11.718280 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.718322 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260925 14:53:11.878815 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.878629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:56061)
0925 14:53:11.878737 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:11.878771 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260925 14:53:12.656172 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.655907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:48503)
0925 14:53:12.656077 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.656125 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260925 14:53:12.719084 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.718840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:45367)
0925 14:53:12.718988 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.719030 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
I20260925 14:53:12.879442 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.879240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:56817)
0925 14:53:12.879360 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:12.879397 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 14:53:13.656910 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.656678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:33033)
0925 14:53:13.656807 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.656850 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 14:53:13.719805 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.719567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:43501)
0925 14:53:13.719724 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.719761 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260925 14:53:13.880144 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.879899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:60381)
0925 14:53:13.880054 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:13.880095 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260925 14:53:14.657732 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.657489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:55739)
0925 14:53:14.657642 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.657682 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260925 14:53:14.720530 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.720331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:56699)
0925 14:53:14.720449 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.720486 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 14:53:14.747429 15698 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:14.747548 15698 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:14.747745 15698 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
W20260925 14:53:14.748257 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
W20260925 14:53:14.748333 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:14.748349 15284 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6589127d13064be788c088a3f69fa096; no voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e
I20260925 14:53:14.748471 15698 raft_consensus.cc:2726] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:53:14.880854 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:14.880641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:60459)
0925 14:53:14.880756 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:14.880793 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 14:53:15.658527 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.658262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:34819)
0925 14:53:15.658436 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.658480 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":61,"threads_started":1}
I20260925 14:53:15.721272 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.721042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:45435)
0925 14:53:15.721181 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.721227 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260925 14:53:15.881691 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.881395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:48841)
0925 14:53:15.881583 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:15.881640 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260925 14:53:16.659327 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.659089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:36381)
0925 14:53:16.659234 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.659280 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260925 14:53:16.722011 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.721775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:46653)
0925 14:53:16.721906 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.721967 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260925 14:53:16.882512 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.882198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:45143)
0925 14:53:16.882415 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:16.882462 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260925 14:53:17.660072 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.659832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:56755)
0925 14:53:17.659985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.660024 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260925 14:53:17.722795 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.722580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:47515)
0925 14:53:17.722712 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.722747 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260925 14:53:17.883241 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.883030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:36323)
0925 14:53:17.883154 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:17.883193 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260925 14:53:18.660810 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.660562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:36013)
0925 14:53:18.660733 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.660765 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260925 14:53:18.723555 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.723275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:59709)
0925 14:53:18.723441 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.723492 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260925 14:53:18.883929 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:18.883709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:45013)
0925 14:53:18.883845 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:18.883883 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260925 14:53:19.461632 15712 raft_consensus.cc:482] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:19.461735 15712 raft_consensus.cc:504] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3915efee0d08492c84356abdcdc1e4b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35043 } } peers { permanent_uuid: "3987721521474cf2b9df1a6b691ba25e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41617 } } peers { permanent_uuid: "6589127d13064be788c088a3f69fa096" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42137 } }
I20260925 14:53:19.461920 15712 leader_election.cc:288] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043), 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617)
W20260925 14:53:19.462384 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3915efee0d08492c84356abdcdc1e4b1 (127.0.98.3:35043): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
W20260925 14:53:19.462469 15284 leader_election.cc:341] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3987721521474cf2b9df1a6b691ba25e (127.0.98.2:41617): Not found: Tablet not found: da89f6f3b2324839847d0320d3ce57ec
I20260925 14:53:19.462513 15284 leader_election.cc:302] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6589127d13064be788c088a3f69fa096; no voters: 3915efee0d08492c84356abdcdc1e4b1, 3987721521474cf2b9df1a6b691ba25e
I20260925 14:53:19.462605 15712 raft_consensus.cc:2726] T da89f6f3b2324839847d0320d3ce57ec P 6589127d13064be788c088a3f69fa096 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:53:19.661556 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.661324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:36621)
0925 14:53:19.661460 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.661505 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260925 14:53:19.724260 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.724037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:53593)
0925 14:53:19.724171 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.724215 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260925 14:53:19.884636 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.884370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:54399)
0925 14:53:19.884531 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:19.884578 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260925 14:53:20.662451 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.662164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.1:60537)
0925 14:53:20.662298 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.662385 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260925 14:53:20.725060 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.724826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.2:52581)
0925 14:53:20.724972 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.725014 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 14:53:20.876606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15237
I20260925 14:53:20.881685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15401
I20260925 14:53:20.885416 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.885182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38245 (local address 127.0.98.3:38977)
0925 14:53:20.885308 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:20.885354 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
I20260925 14:53:20.886386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15533
2026-09-25T14:53:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18212 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:20 dist-test-slave-gpc1 krb5kdc[15721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:20 dist-test-slave-gpc1 krb5kdc[15721](info): set up 2 sockets
Sep 25 14:53:20 dist-test-slave-gpc1 krb5kdc[15721](info): commencing operation
krb5kdc: starting...
W20260925 14:53:21.938530 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:22.057482 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:22.138309 15734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:22.138537 15734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:22.138559 15734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:22.139654 15734 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:53:22.139688 15734 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:53:22.139701 15734 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:53:22.139714 15734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:22.139725 15734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:22.139739 15734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:22.139750 15734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:22.139765 15734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:22.141171 15734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:22.141363 15734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:22.141534 15734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:22.146654 15739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.146710 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.146716 15740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:22.146781 15734 server_base.cc:1034] running on GCE node
I20260925 14:53:22.147178 15734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:22.147500 15734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:22.148665 15734 hybrid_clock.cc:648] HybridClock initialized: now 1790348002148646 us; error 37 us; skew 500 ppm
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:22.151299 15734 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:22.151691 15734 webserver.cc:460] Webserver started at http://127.0.98.62:33477/ using document root <none> and password file <none>
I20260925 14:53:22.151892 15734 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:22.151932 15734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:22.152019 15734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:22.153070 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "d2aeb737939040dfbe2ed5d4304730b8"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "8225d4347958fb74216982bf6bff3da3"
server_key_iv: "bf00d73c500cc0f64122cf024143173a"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.153383 15734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "d2aeb737939040dfbe2ed5d4304730b8"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "8225d4347958fb74216982bf6bff3da3"
server_key_iv: "bf00d73c500cc0f64122cf024143173a"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.154722 15734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:22.155435 15749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.155612 15734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:22.155721 15734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "d2aeb737939040dfbe2ed5d4304730b8"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "8225d4347958fb74216982bf6bff3da3"
server_key_iv: "bf00d73c500cc0f64122cf024143173a"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.155797 15734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:22.174834 15734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:22.177543 15734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:22.177723 15734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:22.181602 15734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45493
I20260925 14:53:22.181653 15801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45493 every 8 connection(s)
I20260925 14:53:22.182019 15734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:22.182404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15734
I20260925 14:53:22.182497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:22.182674 392 external_mini_cluster.cc:1428] Setting key a80ffe1e5372d15e0b43a89541d51789
I20260925 14:53:22.183156 15802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:22.186498 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap starting.
I20260925 14:53:22.187172 15802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:22.187477 15802 log.cc:824] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:22.188313 15802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: No bootstrap required, opened a new log
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:22.190668 15802 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:22.190835 15802 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:22.190886 15802 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Initialized, Role: FOLLOWER
I20260925 14:53:22.191031 15802 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:22.191128 15802 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:22.191167 15802 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:22.191227 15802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:22.191811 15802 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:22.191898 15802 leader_election.cc:302] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2aeb737939040dfbe2ed5d4304730b8; no voters:
I20260925 14:53:22.192142 15802 leader_election.cc:288] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:22.192171 15807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:22.192288 15807 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 LEADER]: Becoming Leader. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Running, Role: LEADER
I20260925 14:53:22.192430 15802 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:22.192410 15807 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:22.192911 15809 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2aeb737939040dfbe2ed5d4304730b8. Latest consensus state: current_term: 1 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:53:22.193002 15809 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:22.193243 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:53:22.193584 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:22.193984 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.183206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.98.62:45493)
0925 14:53:22.183377 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.183385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.183416 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:53:22.183909 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.183915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.183975 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.184087 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.184835 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.185731 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.186792 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.187208 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.187344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.190031 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:22.190041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:22.190041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:22.190053 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:22.191846 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.192303 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.192306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.192307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.192433 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.192649 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.192649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.192649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.192948 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:22.193463 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.193559 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.193707 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260925 14:53:22.193760 15815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:22.194535 15815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:22.196048 15815 catalog_manager.cc:1347] Generated new cluster ID: 11042834a73c4def9fd0c302aac90a44
I20260925 14:53:22.196115 15815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:22.217568 15815 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:22.218405 15815 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:22.227790 15815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Generated new TSK 0
I20260925 14:53:22.228029 15815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:53:22.259428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:22.341746 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:22.341951 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:22.341976 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:22.343058 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:22.343096 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:22.343122 15828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:53:22.344673 15828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:22.344933 15828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:22.345136 15828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:22.350145 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.350162 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:22.350221 15828 server_base.cc:1034] running on GCE node
W20260925 14:53:22.350145 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:22.350663 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:22.350874 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:22.352037 15828 hybrid_clock.cc:648] HybridClock initialized: now 1790348002352020 us; error 34 us; skew 500 ppm
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:22.354679 15828 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:53:22.355218 15828 webserver.cc:460] Webserver started at http://127.0.98.1:36863/ using document root <none> and password file <none>
I20260925 14:53:22.355466 15828 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:22.355533 15828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:22.355687 15828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:22.356734 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "69efcb4590ad6f2f69ed0f175b2fda84"
server_key_iv: "d955da3f4d8c0658d50a5984a824dd12"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.357093 15828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "69efcb4590ad6f2f69ed0f175b2fda84"
server_key_iv: "d955da3f4d8c0658d50a5984a824dd12"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.358440 15828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:22.359194 15843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.359365 15828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:22.359464 15828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "69efcb4590ad6f2f69ed0f175b2fda84"
server_key_iv: "d955da3f4d8c0658d50a5984a824dd12"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.359537 15828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:22.378705 15828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:22.380715 15828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:22.380910 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:22.381172 15828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:22.381503 15828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:22.381558 15828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.381600 15828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:22.381649 15828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.387679 15828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45627
I20260925 14:53:22.387732 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45627 every 8 connection(s)
I20260925 14:53:22.388055 15828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:22.394480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15828
I20260925 14:53:22.394567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:53:22.394757 392 external_mini_cluster.cc:1428] Setting key 43c5e16fba87450543c7253d7105f0ae
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:22.398835 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.389794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45727 (local address 127.0.98.62:45493)
0925 14:53:22.389878 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.389880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.390050 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:53:22.390545 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.390546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.390556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.390608 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.391515 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.392005 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.393061 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.393294 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.393346 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.395816 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:22.395818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:22.395818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:22.395826 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:22.397333 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.397831 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.397832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.397832 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.397880 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.398193 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.398194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.398194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.398291 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:22.398382 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.398699 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.398771 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 14:53:22.399168 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.389746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:45727)
0925 14:53:22.389920 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.389958 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.390462 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.390663 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.390667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.391240 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.391434 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.391438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.392089 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.392090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.392978 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.392981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.393360 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.393361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.393512 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.394104 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.394116 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.395719 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:22.397420 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.397427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.397430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.397755 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.397965 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.397966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.397967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.398129 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.398379 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:22.398385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:22.398562 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.398678 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.398861 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260925 14:53:22.399770 15957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:22.399863 15957 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:22.400063 15957 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:22.400593 15766 ts_manager.cc:194] Registered new tserver with Master: 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1:45627)
I20260925 14:53:22.401541 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45727
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:53:22.421519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:22.508414 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:22.508677 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:22.508718 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:22.509953 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:22.510025 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:22.510084 15962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:53:22.511705 15962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:22.512005 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:22.512267 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:22.517567 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.517700 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.517567 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:22.518064 15962 server_base.cc:1034] running on GCE node
I20260925 14:53:22.518324 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:22.518673 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:22.519883 15962 hybrid_clock.cc:648] HybridClock initialized: now 1790348002519853 us; error 39 us; skew 500 ppm
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:22.523517 15962 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:53:22.524013 15962 webserver.cc:460] Webserver started at http://127.0.98.2:42343/ using document root <none> and password file <none>
I20260925 14:53:22.524257 15962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:22.524328 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:22.524477 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:22.525560 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "7f9c4657aa62d5e1c21eb69d41e24c34"
server_key_iv: "70d0887e5f861feca088a94f9a677c62"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.525964 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "7f9c4657aa62d5e1c21eb69d41e24c34"
server_key_iv: "70d0887e5f861feca088a94f9a677c62"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.527822 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 14:53:22.528592 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.528769 15962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.528872 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "7f9c4657aa62d5e1c21eb69d41e24c34"
server_key_iv: "70d0887e5f861feca088a94f9a677c62"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.528963 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:22.552637 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:22.554773 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:22.554970 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:22.555229 15962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:22.555598 15962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:22.555665 15962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.555711 15962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:22.555738 15962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.561753 15962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44911
I20260925 14:53:22.561832 16090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44911 every 8 connection(s)
I20260925 14:53:22.562219 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:53:22.566721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 15962
I20260925 14:53:22.566831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:53:22.567050 392 external_mini_cluster.cc:1428] Setting key 55b66c7d8048ffcbe8349cb76bc8661e
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:22.572552 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.563018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47377 (local address 127.0.98.62:45493)
0925 14:53:22.563094 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.563096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.563299 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:53:22.563816 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.563817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.563827 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.563888 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.564867 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.565361 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.566420 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.566704 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.566781 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.569295 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:22.569297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:22.569297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:22.569305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:22.571093 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.571507 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.571508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.571509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.571578 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.571850 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.571851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.571851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.571979 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:22.572062 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.572353 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.572443 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:53:22.572767 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.562995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:47377)
0925 14:53:22.563151 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.563195 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.563726 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.563944 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.563948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.564575 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.564775 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.564780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.565450 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.565451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.566292 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.566295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.566810 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.566811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.566962 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.567488 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.567499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.569207 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:22.571187 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.571193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.571195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.571430 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.571649 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.571650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.571650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.571753 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.572071 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:22.572078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:22.572218 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.572329 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.572521 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260925 14:53:22.573310 16091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:22.573428 16091 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:22.573611 16091 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:22.574077 15766 ts_manager.cc:194] Registered new tserver with Master: ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:22.574653 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47377
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:53:22.595250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:22.680708 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:22.680917 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:22.680940 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:22.682107 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:22.682152 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:22.682178 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:22.683657 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:22.683905 16096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:22.684144 16096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:22.689393 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.689383 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:22.689396 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:22.689584 16096 server_base.cc:1034] running on GCE node
I20260925 14:53:22.689836 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:22.690094 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:22.691239 16096 hybrid_clock.cc:648] HybridClock initialized: now 1790348002691221 us; error 38 us; skew 500 ppm
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:22.693997 16096 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:53:22.694559 16096 webserver.cc:460] Webserver started at http://127.0.98.3:45089/ using document root <none> and password file <none>
I20260925 14:53:22.694747 16096 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:22.694788 16096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:22.694878 16096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:22.695919 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "f0355585ffa2448eb735146e1fd2eafc"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "fa13e6376745e239cbf2cba90893e56d"
server_key_iv: "35d3e3c4d35ec69e52f11fc8900923bf"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.696229 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f0355585ffa2448eb735146e1fd2eafc"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "fa13e6376745e239cbf2cba90893e56d"
server_key_iv: "35d3e3c4d35ec69e52f11fc8900923bf"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.697496 16096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:22.698189 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.698417 16096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:22.698503 16096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "f0355585ffa2448eb735146e1fd2eafc"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "fa13e6376745e239cbf2cba90893e56d"
server_key_iv: "35d3e3c4d35ec69e52f11fc8900923bf"
server_key_version: "encryptionkey@0"
I20260925 14:53:22.698570 16096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:22.716353 16096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:22.718271 16096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:22.718468 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:22.718686 16096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:22.719019 16096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:22.719058 16096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.719082 16096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:22.719095 16096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:22.725229 16096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34911
I20260925 14:53:22.725282 16224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34911 every 8 connection(s)
I20260925 14:53:22.725598 16096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:22.731099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16096
I20260925 14:53:22.731206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:22.731426 392 external_mini_cluster.cc:1428] Setting key d039cc1d4d6fc813e1d8e18322b9cf47
Sep 25 14:53:22 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:22.735394 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.726325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60309 (local address 127.0.98.62:45493)
0925 14:53:22.726407 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.726410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.726658 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:53:22.727163 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.727165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.727175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.727228 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.728232 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.728724 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.729705 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.729901 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.729959 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.732635 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:22.732636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:22.732636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:22.732643 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:22.734101 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.734562 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.734563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.734563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.734617 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.734828 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.734829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.734829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.734950 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:22.734998 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.735259 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.735325 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:53:22.735585 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.726294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:60309)
0925 14:53:22.726516 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.726560 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.727055 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.727290 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.727293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.727924 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:22.728135 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.728141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.728809 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.728810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.729620 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.729622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.729979 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.729980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.730081 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.730644 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:22.730652 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:22.732552 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:22.734172 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.734178 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.734180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.734484 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.734677 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:22.734678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:22.734678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:22.734785 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:22.735002 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:22.735006 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:22.735138 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.735218 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.735345 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":143,"threads_started":1}
I20260925 14:53:22.736256 16225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:22.736370 16225 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:22.736573 16225 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:22.736987 15766 ts_manager.cc:194] Registered new tserver with Master: f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911)
I20260925 14:53:22.737390 15766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60309
I20260925 14:53:22.743103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:22.750756 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.744466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.0.98.62:45493)
0925 14:53:22.744538 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.744541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.744654 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:53:22.744733 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.744734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.744751 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.744808 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:22.745386 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.745901 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.746746 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.746936 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.747041 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.748270 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:22.748272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:22.748272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:22.748279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:22.749439 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.749799 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.749801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.749801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.749874 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:22.750096 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:22.750097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:22.750097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:22.750275 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:22.750422 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.750564 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.750649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:22.752323 15766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:22.753993 15766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:22.760805 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60476 (local address 127.0.98.3:34911)
0925 14:53:22.757397 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.757402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.757412 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:22.757447 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.757449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.757463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.757534 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:22.757677 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758774 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.759857 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760360 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760453 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760519 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.760675 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.760716 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":78,"threads_started":1}
I20260925 14:53:22.760793 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34911 (local address 127.0.0.1:60476)
0925 14:53:22.757209 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.757219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.757259 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.757546 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.757547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.757549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:22.757604 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.757606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758850 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.758851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.759769 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.759771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760464 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.760565 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760578 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.760660 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.760716 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":100,"threads_started":1}
I20260925 14:53:22.761109 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44911 (local address 127.0.0.1:59278)
0925 14:53:22.756767 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.756776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.756820 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.757138 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.757139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.757142 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:22.757248 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.757250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758321 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.758322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.759821 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.759825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760833 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.760912 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760915 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.760962 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.761019 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":41,"thread_start_us":89,"threads_started":1}
I20260925 14:53:22.761327 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.0.98.1:45627)
0925 14:53:22.756971 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.756975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.756983 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:22.757014 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.757016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.757028 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.757117 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:22.757300 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758515 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.759928 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760664 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760735 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760794 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.761156 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.761213 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260925 14:53:22.761109 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45627 (local address 127.0.0.1:43100)
0925 14:53:22.756682 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.756723 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.756798 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.757120 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.757123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.757125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:22.757209 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.757212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758601 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.758602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.759823 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.759825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760834 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.760908 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760912 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.760963 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.761019 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":12,"thread_start_us":66,"threads_started":1}
I20260925 14:53:22.761497 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.756770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.0.98.2:44911)
0925 14:53:22.756944 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.756950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.756962 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:22.757005 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.757007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.757019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.757126 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:22.757334 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.758219 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.759927 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.760663 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.760735 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.760794 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.761346 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.761404 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260925 14:53:22.762221 16025 tablet_service.cc:1467] Processing CreateTablet for tablet 64a9768548e64072b95083f22f2b05cb (DEFAULT_TABLE table=test-table [id=3937851e868b4445bd5ea0e692f6747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:22.762221 16159 tablet_service.cc:1467] Processing CreateTablet for tablet 64a9768548e64072b95083f22f2b05cb (DEFAULT_TABLE table=test-table [id=3937851e868b4445bd5ea0e692f6747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:22.762646 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a9768548e64072b95083f22f2b05cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:22.762646 16025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a9768548e64072b95083f22f2b05cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:22.766177 16243 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Bootstrap starting.
I20260925 14:53:22.766510 16242 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Bootstrap starting.
I20260925 14:53:22.766943 15891 tablet_service.cc:1467] Processing CreateTablet for tablet 64a9768548e64072b95083f22f2b05cb (DEFAULT_TABLE table=test-table [id=3937851e868b4445bd5ea0e692f6747d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:22.767076 16243 tablet_bootstrap.cc:654] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:22.767208 16242 tablet_bootstrap.cc:654] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:22.767325 15891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64a9768548e64072b95083f22f2b05cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:22.767781 16243 log.cc:824] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:22.767831 16242 log.cc:824] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:22.768592 16243 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: No bootstrap required, opened a new log
I20260925 14:53:22.768652 16243 ts_tablet_manager.cc:1397] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:53:22.769083 16242 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: No bootstrap required, opened a new log
I20260925 14:53:22.769233 16242 ts_tablet_manager.cc:1397] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:53:22.770376 16246 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap starting.
I20260925 14:53:22.770771 16243 raft_consensus.cc:348] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.770969 16243 raft_consensus.cc:374] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:22.771009 16243 raft_consensus.cc:729] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0355585ffa2448eb735146e1fd2eafc, State: Initialized, Role: FOLLOWER
I20260925 14:53:22.771031 16246 tablet_bootstrap.cc:654] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:22.771184 16243 consensus_queue.cc:260] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.771351 16242 raft_consensus.cc:348] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.771486 16243 ts_tablet_manager.cc:1428] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:53:22.771540 16242 raft_consensus.cc:374] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:22.771613 16225 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260925 14:53:22.771620 16242 raft_consensus.cc:729] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef00f20eef7b418ea4fd5e68c9652a53, State: Initialized, Role: FOLLOWER
I20260925 14:53:22.771647 16246 log.cc:824] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:22.771740 16242 consensus_queue.cc:260] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.772023 16242 ts_tablet_manager.cc:1428] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:53:22.772125 16091 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260925 14:53:22.772378 16246 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: No bootstrap required, opened a new log
I20260925 14:53:22.772466 16246 ts_tablet_manager.cc:1397] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:22.774096 16246 raft_consensus.cc:348] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.774237 16246 raft_consensus.cc:374] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:22.774259 16246 raft_consensus.cc:729] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Initialized, Role: FOLLOWER
I20260925 14:53:22.774420 16246 consensus_queue.cc:260] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.774701 16246 ts_tablet_manager.cc:1428] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:22.774748 15957 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
W20260925 14:53:22.812840 16092 tablet.cc:2369] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:22.855844 16250 raft_consensus.cc:482] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:22.855974 16250 raft_consensus.cc:504] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.856426 16250 leader_election.cc:288] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:22.859426 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.856713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58869 (local address 127.0.98.3:34911)
0925 14:53:22.856765 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.856767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.856780 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:53:22.856856 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.856857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.856866 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.856939 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:22.857112 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.857611 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.858667 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.859059 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.859168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.859178 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.859323 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.859361 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:53:22.859426 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.856643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34911 (local address 127.0.98.1:58869)
0925 14:53:22.856719 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.856732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.856785 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.856946 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.856948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.856950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:22.857025 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.857039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.857701 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.857702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.858586 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.858589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.859170 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.859171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.859251 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.859254 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.859311 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.859351 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 14:53:22.859814 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a9768548e64072b95083f22f2b05cb" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0355585ffa2448eb735146e1fd2eafc" is_pre_election: true
I20260925 14:53:22.859984 16179 raft_consensus.cc:2454] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 0.
I20260925 14:53:22.859989 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.856730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53187 (local address 127.0.98.2:44911)
0925 14:53:22.856780 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.856782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.857055 (+ 273us) server_negotiation.cc:407] Connection header received
0925 14:53:22.857118 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.857120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.857133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.857207 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:22.857364 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.857823 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.859029 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.859631 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.859720 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.859730 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.859886 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.859926 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:53:22.860005 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.856645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44911 (local address 127.0.98.1:53187)
0925 14:53:22.856985 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.856993 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:22.857063 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:22.857210 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:22.857211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:22.857213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:22.857288 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.857290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.857923 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.857924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.858939 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:22.858942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.859723 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.859724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:22.859795 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.859798 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:22.859880 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:53:22.859925 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":112,"thread_start_us":123,"threads_started":1}
I20260925 14:53:22.860236 15845 leader_election.cc:302] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6, f0355585ffa2448eb735146e1fd2eafc; no voters:
I20260925 14:53:22.860375 16250 raft_consensus.cc:2781] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:22.860435 16250 raft_consensus.cc:482] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:22.860375 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a9768548e64072b95083f22f2b05cb" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" is_pre_election: true
I20260925 14:53:22.860469 16250 raft_consensus.cc:3037] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:22.860538 16045 raft_consensus.cc:2454] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 0.
I20260925 14:53:22.861279 16250 raft_consensus.cc:504] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.861440 16250 leader_election.cc:288] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 election: Requested vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:22.861583 16179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a9768548e64072b95083f22f2b05cb" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0355585ffa2448eb735146e1fd2eafc"
I20260925 14:53:22.861593 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64a9768548e64072b95083f22f2b05cb" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
I20260925 14:53:22.861660 16179 raft_consensus.cc:3037] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:22.861685 16045 raft_consensus.cc:3037] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:22.862504 16179 raft_consensus.cc:2454] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 1.
I20260925 14:53:22.862543 16045 raft_consensus.cc:2454] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 1.
I20260925 14:53:22.862707 15847 leader_election.cc:302] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6, ef00f20eef7b418ea4fd5e68c9652a53; no voters:
I20260925 14:53:22.862814 16250 raft_consensus.cc:2781] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:22.862957 16250 raft_consensus.cc:686] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 LEADER]: Becoming Leader. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Running, Role: LEADER
I20260925 14:53:22.863075 16250 consensus_queue.cc:237] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:22.863989 15766 catalog_manager.cc:5495] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } health_report { overall_health: UNKNOWN } } }
I20260925 14:53:22.881717 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.878869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.0.98.1:45627)
0925 14:53:22.878916 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:22.878918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:22.878930 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:22.878979 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:22.878980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:22.879004 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:22.879092 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:22.879251 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.879744 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.880537 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:22.880752 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:22.880846 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:22.881064 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:22.881228 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:22.881578 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:22.881600 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 14:53:22.881624 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:22.883706 16045 raft_consensus.cc:1261] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Refusing update from remote peer 4cbd1cbfb1cc4e1286638c624ebe64d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:22.883715 16179 raft_consensus.cc:1261] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Refusing update from remote peer 4cbd1cbfb1cc4e1286638c624ebe64d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:22.884189 16252 consensus_queue.cc:1035] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:22.884400 16250 consensus_queue.cc:1035] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:22.887354 16255 mvcc.cc:205] Tried to move back new op lower bound from 7333265419808866304 to 7333265419727986688. Current Snapshot: MvccSnapshot[applied={T|T < 7333265419808866304 or (T in {7333265419808866304})}]
I20260925 14:53:22.889312 15766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:22.889408 15766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:22.890218 15766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Sending DeleteTablet for 3 replicas of tablet 64a9768548e64072b95083f22f2b05cb
I20260925 14:53:22.890625 16159 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a9768548e64072b95083f22f2b05cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60476
I20260925 14:53:22.890610 16025 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a9768548e64072b95083f22f2b05cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59278
I20260925 14:53:22.890851 15891 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a9768548e64072b95083f22f2b05cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43100
I20260925 14:53:22.890873 16265 tablet_replica.cc:330] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: stopping tablet replica
I20260925 14:53:22.890980 16265 raft_consensus.cc:2229] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:22.891052 16267 tablet_replica.cc:330] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: stopping tablet replica
I20260925 14:53:22.891078 16265 raft_consensus.cc:2258] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:22.891139 16267 raft_consensus.cc:2229] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:22.891142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15828
I20260925 14:53:22.891984 16265 ts_tablet_manager.cc:1905] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:22.893378 16266 tablet_replica.cc:330] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: stopping tablet replica
I20260925 14:53:22.893478 16266 raft_consensus.cc:2229] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:22.893608 16266 raft_consensus.cc:2258] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:22.893615 16265 ts_tablet_manager.cc:1918] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:22.893679 16265 log.cc:1196] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/64a9768548e64072b95083f22f2b05cb
I20260925 14:53:22.894052 16265 ts_tablet_manager.cc:1939] T 64a9768548e64072b95083f22f2b05cb P f0355585ffa2448eb735146e1fd2eafc: Deleting consensus metadata
I20260925 14:53:22.894095 16266 ts_tablet_manager.cc:1905] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:22.894539 15751 catalog_manager.cc:4841] TS f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): tablet 64a9768548e64072b95083f22f2b05cb (table test-table [id=3937851e868b4445bd5ea0e692f6747d]) successfully deleted
I20260925 14:53:22.895589 16266 ts_tablet_manager.cc:1918] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:22.895650 16266 log.cc:1196] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/64a9768548e64072b95083f22f2b05cb
I20260925 14:53:22.895913 16266 ts_tablet_manager.cc:1939] T 64a9768548e64072b95083f22f2b05cb P ef00f20eef7b418ea4fd5e68c9652a53: Deleting consensus metadata
I20260925 14:53:22.896302 15753 catalog_manager.cc:4841] TS ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): tablet 64a9768548e64072b95083f22f2b05cb (table test-table [id=3937851e868b4445bd5ea0e692f6747d]) successfully deleted
W20260925 14:53:22.897805 15750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45627 (error 108)
I20260925 14:53:22.898372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15962
I20260925 14:53:22.898414 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:22.898240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45627 (local address 127.0.0.1:43118)
0925 14:53:22.898297 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:22.898380 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:53:22.899781 15750 catalog_manager.cc:4584] TS 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1:45627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64a9768548e64072b95083f22f2b05cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45627: connect: Connection refused (error 111)
I20260925 14:53:22.903429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16096
I20260925 14:53:22.908653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 15734
I20260925 14:53:42.914166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--webserver_interface=127.0.98.62
--webserver_port=33477
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:43.002240 16269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:43.002492 16269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:43.002517 16269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:43.003756 16269 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:53:43.003798 16269 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:53:43.003810 16269 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:53:43.003822 16269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:43.003835 16269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:43.003892 16269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:43.003909 16269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:43.003932 16269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:43.005470 16269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:43.005800 16269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:43.006078 16269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:43.015230 16269 server_base.cc:1034] running on GCE node
W20260925 14:53:43.015195 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.015199 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.015195 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:43.015688 16269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:43.016031 16269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:43.017200 16269 hybrid_clock.cc:648] HybridClock initialized: now 1790348023017175 us; error 49 us; skew 500 ppm
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:43.023705 16269 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:43.024184 16269 webserver.cc:460] Webserver started at http://127.0.98.62:33477/ using document root <none> and password file <none>
I20260925 14:53:43.024441 16269 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:43.024525 16269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:43.025978 16269 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:43.026736 16285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.026943 16269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:43.027051 16269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "d2aeb737939040dfbe2ed5d4304730b8"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "8225d4347958fb74216982bf6bff3da3"
server_key_iv: "bf00d73c500cc0f64122cf024143173a"
server_key_version: "encryptionkey@0"
I20260925 14:53:43.027484 16269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:43.054862 16269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:43.057139 16269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:43.057318 16269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:43.061512 16269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45493
I20260925 14:53:43.061555 16337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45493 every 8 connection(s)
I20260925 14:53:43.061975 16269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:43.063627 16338 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:53:43.064329 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap starting.
I20260925 14:53:43.069885 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16269
I20260925 14:53:43.070245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45627
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=36863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:43.073897 16338 log.cc:824] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:43.076046 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:43.076349 16338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap complete.
I20260925 14:53:43.078780 16338 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:43.079021 16338 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.079193 16338 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:43.079253 16338 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:43.079319 16338 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:43.079372 16338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:53:43.080410 16338 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:43.080494 16338 leader_election.cc:302] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2aeb737939040dfbe2ed5d4304730b8; no voters:
I20260925 14:53:43.080713 16338 leader_election.cc:288] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:53:43.080754 16342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:53:43.080883 16342 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 LEADER]: Becoming Leader. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Running, Role: LEADER
I20260925 14:53:43.080991 16338 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:43.080965 16342 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:53:43.081261 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2aeb737939040dfbe2ed5d4304730b8. Latest consensus state: current_term: 2 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:53:43.081339 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:43.081266 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:53:43.081506 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:43.081530 16347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:43.082535 16347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:43.082667 16347 catalog_manager.cc:1259] Loaded cluster ID: 11042834a73c4def9fd0c302aac90a44
I20260925 14:53:43.082940 16347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:43.083922 16347 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:43.084121 16347 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Loaded TSK: 0
I20260925 14:53:43.090819 16347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Generated new TSK 1
I20260925 14:53:43.090903 16347 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:53:43.167196 16340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:43.167466 16340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:43.167510 16340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:43.168691 16340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:43.168766 16340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:43.168812 16340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:53:43.170284 16340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36863
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:43.170567 16340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:43.170818 16340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:43.176733 16366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.176752 16365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.176757 16368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:43.177016 16340 server_base.cc:1034] running on GCE node
I20260925 14:53:43.177233 16340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:43.177492 16340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:43.178655 16340 hybrid_clock.cc:648] HybridClock initialized: now 1790348023178614 us; error 57 us; skew 500 ppm
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:43.181450 16340 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:53:43.181995 16340 webserver.cc:460] Webserver started at http://127.0.98.1:36863/ using document root <none> and password file <none>
I20260925 14:53:43.182256 16340 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:43.182374 16340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:43.184165 16340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:43.184947 16375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.185146 16340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:43.185237 16340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "69efcb4590ad6f2f69ed0f175b2fda84"
server_key_iv: "d955da3f4d8c0658d50a5984a824dd12"
server_key_version: "encryptionkey@0"
I20260925 14:53:43.185681 16340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:43.209738 16340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:43.212496 16340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:43.212661 16340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:43.212939 16340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:43.213451 16382 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:53:43.215085 16340 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:53:43.215149 16340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:53:43.215186 16340 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:53:43.216054 16340 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:53:43.216100 16340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:43.216210 16382 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap starting.
I20260925 14:53:43.223003 16340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45627
I20260925 14:53:43.223296 16489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45627 every 8 connection(s)
I20260925 14:53:43.223443 16340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:53:43.225986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16340
I20260925 14:53:43.226279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44911
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=42343
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:43.227780 16382 log.cc:824] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:43.230561 16382 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:53:43.230885 16382 tablet_bootstrap.cc:492] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap complete.
I20260925 14:53:43.232045 16382 ts_tablet_manager.cc:1397] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260925 14:53:43.233134 16382 raft_consensus.cc:348] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
I20260925 14:53:43.233501 16382 raft_consensus.cc:729] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.233666 16382 consensus_queue.cc:260] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } } peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } }
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:43.233945 16382 ts_tablet_manager.cc:1428] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:53:43.238592 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.224343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46561 (local address 127.0.98.62:45493)
0925 14:53:43.225709 (+ 1366us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.225719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.225764 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:53:43.225923 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.225930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.226004 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.226248 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.227607 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.229118 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.230805 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.231516 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.231622 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.234368 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.234372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.234373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.234385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.236499 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.237047 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.237050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.237050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.237155 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.237457 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.237457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.237458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.237722 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.237806 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.238212 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.238396 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":102,"threads_started":1}
I20260925 14:53:43.238605 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.224221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:46561)
0925 14:53:43.224599 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.224664 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.225458 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.226636 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.226657 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.227142 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:43.227462 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.227468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.229257 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.229258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.230651 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.230654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.231634 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.231636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.231785 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.232399 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:43.232410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:43.234232 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:43.236585 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.236593 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.236595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.236957 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.237230 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.237231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.237232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.237386 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.237799 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:43.237806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:43.237984 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.238118 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.238321 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":94,"threads_started":1}
I20260925 14:53:43.240804 16490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:43.240911 16490 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:43.241101 16490 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:43.241899 16302 ts_manager.cc:194] Registered new tserver with Master: 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1:45627)
I20260925 14:53:43.243220 16302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46561
I20260925 14:53:43.244428 16490 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:53:43.254288 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.242694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.98.1:45627)
0925 14:53:43.242842 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.242846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.243069 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:53:43.243259 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.243261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.243277 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.243323 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.243937 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.245028 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.245870 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.246396 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.246469 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.251301 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.251309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.251310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.251317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.253057 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.253498 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.253500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.253501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.253553 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.253752 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.253753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.253753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.253892 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.253980 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.254148 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.254222 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 14:53:43.254374 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.242636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45627 (local address 127.0.0.1:48968)
0925 14:53:43.242911 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.242947 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.243124 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.243415 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.243418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.243777 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:43.243842 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.243844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.245116 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.245117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.245769 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.245771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.246484 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.246487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.246582 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.249599 (+ 3017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:43.249604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:43.251195 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:43.253142 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.253149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.253152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.253416 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.253618 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.253619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.253619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.253694 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.253970 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:43.253977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:43.254033 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.254125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.254248 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
I20260925 14:53:43.254793 16424 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a9768548e64072b95083f22f2b05cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48968
I20260925 14:53:43.255391 16499 tablet_replica.cc:330] stopping tablet replica
I20260925 14:53:43.255517 16499 raft_consensus.cc:2229] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.255610 16499 raft_consensus.cc:2258] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.255934 16499 ts_tablet_manager.cc:1905] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.257922 16499 ts_tablet_manager.cc:1918] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:43.258014 16499 log.cc:1196] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal/wals/64a9768548e64072b95083f22f2b05cb
I20260925 14:53:43.258225 16499 ts_tablet_manager.cc:1939] T 64a9768548e64072b95083f22f2b05cb P 4cbd1cbfb1cc4e1286638c624ebe64d6: Deleting consensus metadata
I20260925 14:53:43.258785 16286 catalog_manager.cc:4841] TS 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1:45627): tablet 64a9768548e64072b95083f22f2b05cb (table test-table [id=3937851e868b4445bd5ea0e692f6747d]) successfully deleted
W20260925 14:53:43.321668 16494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:43.321868 16494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:43.321892 16494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:43.323074 16494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:43.323114 16494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:43.323171 16494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:53:43.324592 16494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42343
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:43.324829 16494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:43.325060 16494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:43.330013 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.329969 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:43.330070 16494 server_base.cc:1034] running on GCE node
W20260925 14:53:43.329982 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:43.330436 16494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:43.330710 16494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:43.331874 16494 hybrid_clock.cc:648] HybridClock initialized: now 1790348023331849 us; error 57 us; skew 500 ppm
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:43.334301 16494 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:53:43.334793 16494 webserver.cc:460] Webserver started at http://127.0.98.2:42343/ using document root <none> and password file <none>
I20260925 14:53:43.335018 16494 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:43.335099 16494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:43.336503 16494 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:43.337126 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.337297 16494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:43.337405 16494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "7f9c4657aa62d5e1c21eb69d41e24c34"
server_key_iv: "70d0887e5f861feca088a94f9a677c62"
server_key_version: "encryptionkey@0"
I20260925 14:53:43.337754 16494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:43.349202 16494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:43.351246 16494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:43.351415 16494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:43.351661 16494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:43.351960 16494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:43.352013 16494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.352052 16494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:43.352080 16494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.357959 16494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44911
I20260925 14:53:43.357996 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44911 every 8 connection(s)
I20260925 14:53:43.358448 16494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:43.367268 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.359200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54173 (local address 127.0.98.62:45493)
0925 14:53:43.359279 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.359281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.359488 (+ 207us) server_negotiation.cc:407] Connection header received
0925 14:53:43.359998 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.359999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.360012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.360063 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.360958 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.361443 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.362474 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.362675 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.362728 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.364724 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.364725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.364725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.364732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.366137 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.366526 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.366527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.366527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.366574 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.366773 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.366774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.366774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.366877 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.366924 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.367125 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.367205 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:53:43.367398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16494
I20260925 14:53:43.367527 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.359163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:54173)
0925 14:53:43.359342 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.359377 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.359913 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.360116 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.360119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.360694 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:43.360885 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.360890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.361519 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.361520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.362319 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.362321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.362745 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.362746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.362847 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.363303 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:43.363310 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:43.364610 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:43.366203 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.366208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.366210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.366477 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.366631 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.366631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.366631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.366733 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.366929 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:43.366932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:43.367039 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.367115 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.367270 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260925 14:53:43.367671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34911
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=45089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:43.368009 16628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:43.368109 16628 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:43.368296 16628 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:43.368680 16302 ts_manager.cc:194] Registered new tserver with Master: ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:43.369128 16302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54173
W20260925 14:53:43.448007 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:43.448210 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:43.448235 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:43.449306 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:43.449338 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:43.449363 16631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:43.450726 16631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45089
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:53:43.450917 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:43.451098 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:43.456207 16631 server_base.cc:1034] running on GCE node
W20260925 14:53:43.456192 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.456357 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:43.456192 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:43.456635 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:43.456907 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:43.458053 16631 hybrid_clock.cc:648] HybridClock initialized: now 1790348023458039 us; error 27 us; skew 500 ppm
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:43.460484 16631 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:53:43.460975 16631 webserver.cc:460] Webserver started at http://127.0.98.3:45089/ using document root <none> and password file <none>
I20260925 14:53:43.461197 16631 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:43.461277 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:43.462694 16631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:43.463265 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.463435 16631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:43.463536 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "f0355585ffa2448eb735146e1fd2eafc"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "fa13e6376745e239cbf2cba90893e56d"
server_key_iv: "35d3e3c4d35ec69e52f11fc8900923bf"
server_key_version: "encryptionkey@0"
I20260925 14:53:43.463918 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:43.494138 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:43.496235 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:43.496414 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:43.496687 16631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:43.497007 16631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:43.497061 16631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.497097 16631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:43.497143 16631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:43.503234 16631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34911
I20260925 14:53:43.503299 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34911 every 8 connection(s)
I20260925 14:53:43.503688 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:53:43 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:43.513240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16631
I20260925 14:53:43.513340 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.504540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59411 (local address 127.0.98.62:45493)
0925 14:53:43.504647 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.504650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.504845 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:53:43.505399 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.505400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.505413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.505456 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.506554 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.507052 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.508088 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.508304 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.508390 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.510677 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.510678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.510679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.510685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.511927 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.512351 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.512352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.512353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.512411 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.512683 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.512685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.512685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.512798 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.512874 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.513153 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.513250 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:53:43.513542 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.504520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:59411)
0925 14:53:43.504704 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.504750 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.505286 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.505547 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.505551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.506183 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:43.506454 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.506460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.507151 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.507153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.507989 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.507991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.508396 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.508397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.508532 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.509043 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:43.509052 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:43.510538 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:43.512012 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.512019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.512021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.512258 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.512498 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:43.512499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:43.512499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:43.512606 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:43.512864 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:43.512871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:43.513014 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.513127 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.513285 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260925 14:53:43.514035 16760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45493
I20260925 14:53:43.514156 16760 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:43.514413 16760 heartbeater.cc:511] Master 127.0.98.62:45493 requested a full tablet report, sending...
I20260925 14:53:43.514878 16302 ts_manager.cc:194] Registered new tserver with Master: f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911)
I20260925 14:53:43.515265 16302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59411
I20260925 14:53:43.520157 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.513740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40514 (local address 127.0.98.62:45493)
0925 14:53:43.513779 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.513781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.514052 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:53:43.514149 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.514150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.514179 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.514237 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.515003 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.515525 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.516319 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.516520 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.516602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.517819 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.517820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.517820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.517827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.519054 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.519403 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.519404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.519405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.519461 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.519657 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.519658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.519658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.519768 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.519949 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.519992 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.520059 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:43.520812 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:43.523289 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.521179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.0.98.62:45493)
0925 14:53:43.521214 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.521215 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.521325 (+ 110us) server_negotiation.cc:407] Connection header received
0925 14:53:43.521370 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.521371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.521397 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.521453 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:43.521552 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.521979 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.522751 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.522919 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.522972 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.523090 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.523141 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 14:53:43.523250 (+ 109us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40516: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:53:43.523352 16493 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40516: token expired
W20260925 14:53:43.523476 16230 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45493)
I20260925 14:53:43.528945 16493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.523743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.0.98.62:45493)
0925 14:53:43.523773 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.523774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.523784 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:43.523848 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.523849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.523855 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.523890 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:43.524293 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.524718 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.525362 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.525534 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.525594 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.526874 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:43.526875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:43.526875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:43.526881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:43.528086 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.528394 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.528395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.528395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.528442 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:43.528576 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:43.528577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:43.528577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:43.528674 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:43.528746 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.528837 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.528889 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 14:53:43.530144 16302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:43.531916 16302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:43.533998 16424 tablet_service.cc:1467] Processing CreateTablet for tablet 24927187f30e449894c23df664d098b4 (DEFAULT_TABLE table=test-table [id=f2da1aea97634a3081725be3baa46923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.534214 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24927187f30e449894c23df664d098b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.535658 16382 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap starting.
I20260925 14:53:43.536026 16382 tablet_bootstrap.cc:654] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.536639 16382 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: No bootstrap required, opened a new log
I20260925 14:53:43.536804 16382 ts_tablet_manager.cc:1397] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:43.537177 16382 raft_consensus.cc:348] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.537272 16382 raft_consensus.cc:374] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.537324 16382 raft_consensus.cc:729] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.537370 16382 consensus_queue.cc:260] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.537456 16382 ts_tablet_manager.cc:1428] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:43.538254 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.533712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34911 (local address 127.0.0.1:39082)
0925 14:53:43.533759 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.533770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.533806 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.534283 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.534284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.534286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.534336 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.534358 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.535769 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.535770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.537038 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.537040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.537957 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.537958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.538030 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.538050 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.538124 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.538168 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":32}
I20260925 14:53:43.538383 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.533863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.98.3:34911)
0925 14:53:43.534071 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.534076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.534089 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:53:43.534143 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.534145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.534167 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.534437 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.534530 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.535664 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.537125 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.537874 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.537947 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.538009 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.538134 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.538204 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":129,"threads_started":1}
I20260925 14:53:43.538465 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.533998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.98.2:44911)
0925 14:53:43.534281 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.534284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.534293 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:43.534375 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.534377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.534395 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.534611 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.534694 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.535724 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.537196 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.537947 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.538015 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.538073 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.538294 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.538364 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":90,"threads_started":1}
I20260925 14:53:43.538563 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.533757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44911 (local address 127.0.0.1:35416)
0925 14:53:43.534078 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.534089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.534141 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.534493 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.534494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.534496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.534541 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.534542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.535809 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.535810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.537079 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.537083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.538107 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.538108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.538170 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.538173 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.538219 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.538248 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":67,"thread_start_us":71,"threads_started":1}
I20260925 14:53:43.539099 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 24927187f30e449894c23df664d098b4 (DEFAULT_TABLE table=test-table [id=f2da1aea97634a3081725be3baa46923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.539172 16562 tablet_service.cc:1467] Processing CreateTablet for tablet 24927187f30e449894c23df664d098b4 (DEFAULT_TABLE table=test-table [id=f2da1aea97634a3081725be3baa46923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:43.539423 16562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24927187f30e449894c23df664d098b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.539434 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24927187f30e449894c23df664d098b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:43.542271 16769 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Bootstrap starting.
I20260925 14:53:43.542469 16770 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Bootstrap starting.
I20260925 14:53:43.543004 16769 tablet_bootstrap.cc:654] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.543068 16770 tablet_bootstrap.cc:654] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:43.543515 16769 log.cc:824] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:43.543694 16770 log.cc:824] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:43.544317 16769 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: No bootstrap required, opened a new log
I20260925 14:53:43.544389 16769 ts_tablet_manager.cc:1397] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:43.544661 16770 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: No bootstrap required, opened a new log
I20260925 14:53:43.544742 16770 ts_tablet_manager.cc:1397] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:43.545794 16769 raft_consensus.cc:348] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.545939 16769 raft_consensus.cc:374] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.545961 16769 raft_consensus.cc:729] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0355585ffa2448eb735146e1fd2eafc, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.546083 16769 consensus_queue.cc:260] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.546206 16770 raft_consensus.cc:348] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.546391 16769 ts_tablet_manager.cc:1428] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:53:43.546396 16770 raft_consensus.cc:374] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:43.546437 16760 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260925 14:53:43.546438 16770 raft_consensus.cc:729] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef00f20eef7b418ea4fd5e68c9652a53, State: Initialized, Role: FOLLOWER
I20260925 14:53:43.546546 16770 consensus_queue.cc:260] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.546829 16628 heartbeater.cc:503] Master 127.0.98.62:45493 was elected leader, sending a full tablet report...
I20260925 14:53:43.547071 16770 ts_tablet_manager.cc:1428] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260925 14:53:43.609083 16629 tablet.cc:2369] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:43.659229 16496 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:43.659407 16496 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.659775 16496 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:43.662737 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.660029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38297 (local address 127.0.98.2:44911)
0925 14:53:43.660090 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.660092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.660102 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:43.660183 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.660184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.660201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.660279 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.660443 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.660906 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.661923 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.662297 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.662433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.662446 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.662624 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.662667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:53:43.662739 16492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.659973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44911 (local address 127.0.98.1:38297)
0925 14:53:43.660029 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.660041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.660095 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.660287 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.660288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.660290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.660364 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.660366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.660986 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.660987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.661827 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.661829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.662438 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.662439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.662521 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.662539 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.662616 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.662659 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":22}
I20260925 14:53:43.662817 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.660023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53135 (local address 127.0.98.3:34911)
0925 14:53:43.660068 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.660070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.660234 (+ 164us) server_negotiation.cc:407] Connection header received
0925 14:53:43.660304 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.660316 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.660328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.660398 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:43.660558 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.661007 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.661986 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.662456 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.662534 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.662544 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.662706 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.662748 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:43.663055 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.659977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34911 (local address 127.0.98.1:53135)
0925 14:53:43.660149 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.660165 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:43.660245 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:43.660400 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:43.660401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:43.660403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:43.660479 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.660481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.661090 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.661091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.661883 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:43.661885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.662535 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.662536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:43.662616 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.662619 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:43.662688 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:53:43.662727 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":30,"thread_start_us":111,"threads_started":1}
I20260925 14:53:43.663110 16582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24927187f30e449894c23df664d098b4" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" is_pre_election: true
I20260925 14:53:43.663275 16582 raft_consensus.cc:2454] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 0.
I20260925 14:53:43.663406 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24927187f30e449894c23df664d098b4" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0355585ffa2448eb735146e1fd2eafc" is_pre_election: true
I20260925 14:53:43.663559 16714 raft_consensus.cc:2454] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 0.
I20260925 14:53:43.663540 16379 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6, ef00f20eef7b418ea4fd5e68c9652a53; no voters:
I20260925 14:53:43.663658 16496 raft_consensus.cc:2781] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:43.663723 16496 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:43.663761 16496 raft_consensus.cc:3037] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.664469 16496 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.664623 16496 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 election: Requested vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
I20260925 14:53:43.664798 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24927187f30e449894c23df664d098b4" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0355585ffa2448eb735146e1fd2eafc"
I20260925 14:53:43.664800 16582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24927187f30e449894c23df664d098b4" candidate_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
I20260925 14:53:43.664875 16714 raft_consensus.cc:3037] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.664892 16582 raft_consensus.cc:3037] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:43.665571 16714 raft_consensus.cc:2454] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 1.
I20260925 14:53:43.665683 16582 raft_consensus.cc:2454] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbd1cbfb1cc4e1286638c624ebe64d6 in term 1.
I20260925 14:53:43.665747 16377 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6, f0355585ffa2448eb735146e1fd2eafc; no voters:
I20260925 14:53:43.665858 16496 raft_consensus.cc:2781] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:43.666016 16496 raft_consensus.cc:686] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 LEADER]: Becoming Leader. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Running, Role: LEADER
I20260925 14:53:43.666107 16496 consensus_queue.cc:237] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:53:43.666924 16302 catalog_manager.cc:5495] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } health_report { overall_health: HEALTHY } } }
I20260925 14:53:43.684242 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.682045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.0.98.1:45627)
0925 14:53:43.682087 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:43.682089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:43.682119 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:53:43.682187 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:43.682188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:43.682205 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:43.682266 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:43.682435 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.682940 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.683610 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:43.683802 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:43.683861 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:43.683982 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:43.684079 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:43.684132 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:43.684167 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:53:43.684184 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:43.685796 16714 raft_consensus.cc:1261] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Refusing update from remote peer 4cbd1cbfb1cc4e1286638c624ebe64d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:43.685806 16582 raft_consensus.cc:1261] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Refusing update from remote peer 4cbd1cbfb1cc4e1286638c624ebe64d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:43.686162 16496 consensus_queue.cc:1035] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:43.686323 16776 consensus_queue.cc:1035] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:43.689519 16780 mvcc.cc:205] Tried to move back new op lower bound from 7333265505014427648 to 7333265504936718336. Current Snapshot: MvccSnapshot[applied={T|T < 7333265505014427648}]
I20260925 14:53:43.689568 16779 mvcc.cc:205] Tried to move back new op lower bound from 7333265505014427648 to 7333265504936718336. Current Snapshot: MvccSnapshot[applied={T|T < 7333265505014427648 or (T in {7333265505014427648})}]
I20260925 14:53:43.690634 16301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.690730 16301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:43.691432 16301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Sending DeleteTablet for 3 replicas of tablet 24927187f30e449894c23df664d098b4
I20260925 14:53:43.691723 16694 tablet_service.cc:1514] Processing DeleteTablet for tablet 24927187f30e449894c23df664d098b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39082
I20260925 14:53:43.691833 16562 tablet_service.cc:1514] Processing DeleteTablet for tablet 24927187f30e449894c23df664d098b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35416
I20260925 14:53:43.691915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16340
I20260925 14:53:43.691939 16424 tablet_service.cc:1514] Processing DeleteTablet for tablet 24927187f30e449894c23df664d098b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48968
I20260925 14:53:43.692030 16499 tablet_replica.cc:330] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: stopping tablet replica
I20260925 14:53:43.692067 16787 tablet_replica.cc:330] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: stopping tablet replica
I20260925 14:53:43.692095 16499 raft_consensus.cc:2229] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:53:43.692167 16499 raft_consensus.cc:2258] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.692173 16787 raft_consensus.cc:2229] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.692273 16787 raft_consensus.cc:2258] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.692272 16788 tablet_replica.cc:330] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: stopping tablet replica
I20260925 14:53:43.692467 16788 raft_consensus.cc:2229] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:43.692611 16788 raft_consensus.cc:2258] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:43.692777 16787 ts_tablet_manager.cc:1905] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.693104 16788 ts_tablet_manager.cc:1905] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:43.694480 16787 ts_tablet_manager.cc:1918] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:43.694558 16787 log.cc:1196] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/24927187f30e449894c23df664d098b4
I20260925 14:53:43.694880 16787 ts_tablet_manager.cc:1939] T 24927187f30e449894c23df664d098b4 P f0355585ffa2448eb735146e1fd2eafc: Deleting consensus metadata
I20260925 14:53:43.695082 16788 ts_tablet_manager.cc:1918] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:43.695181 16788 log.cc:1196] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/24927187f30e449894c23df664d098b4
I20260925 14:53:43.695498 16788 ts_tablet_manager.cc:1939] T 24927187f30e449894c23df664d098b4 P ef00f20eef7b418ea4fd5e68c9652a53: Deleting consensus metadata
I20260925 14:53:43.695612 16287 catalog_manager.cc:4841] TS f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): tablet 24927187f30e449894c23df664d098b4 (table test-table [id=f2da1aea97634a3081725be3baa46923]) successfully deleted
I20260925 14:53:43.696053 16289 catalog_manager.cc:4841] TS ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): tablet 24927187f30e449894c23df664d098b4 (table test-table [id=f2da1aea97634a3081725be3baa46923]) successfully deleted
W20260925 14:53:43.697968 16286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45627 (error 108)
I20260925 14:53:43.698549 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:43.698412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45627 (local address 127.0.0.1:48992)
0925 14:53:43.698474 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:43.698511 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:53:43.698596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16494
W20260925 14:53:43.698706 16286 catalog_manager.cc:4584] TS 4cbd1cbfb1cc4e1286638c624ebe64d6 (127.0.98.1:45627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24927187f30e449894c23df664d098b4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45627: connect: Connection refused (error 111)
I20260925 14:53:43.703768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16631
I20260925 14:53:43.708426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16269
I20260925 14:54:03.714036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--webserver_interface=127.0.98.62
--webserver_port=33477
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:03.794883 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:03.795114 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:03.795151 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:03.796312 16790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:54:03.796379 16790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:54:03.796401 16790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:54:03.796418 16790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:03.796452 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:03.796478 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:03.796504 16790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:03.796542 16790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:03.798030 16790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:03.798305 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:03.798568 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:03.803706 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:03.803828 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:03.803735 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:03.803972 16790 server_base.cc:1034] running on GCE node
I20260925 14:54:03.804193 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:03.804474 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:03.805620 16790 hybrid_clock.cc:648] HybridClock initialized: now 1790348043805601 us; error 35 us; skew 500 ppm
Sep 25 14:54:03 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:03.812994 16790 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:03.813469 16790 webserver.cc:460] Webserver started at http://127.0.98.62:33477/ using document root <none> and password file <none>
I20260925 14:54:03.813673 16790 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:03.813714 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:03.815083 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:03.815696 16805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:03.815871 16790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:03.815941 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "d2aeb737939040dfbe2ed5d4304730b8"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "8225d4347958fb74216982bf6bff3da3"
server_key_iv: "bf00d73c500cc0f64122cf024143173a"
server_key_version: "encryptionkey@0"
I20260925 14:54:03.816272 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:03.834035 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:03.835917 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:03.836026 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:03.840214 16790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45493
I20260925 14:54:03.840253 16857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45493 every 8 connection(s)
I20260925 14:54:03.840653 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:03.842099 16858 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:54:03.842801 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap starting.
I20260925 14:54:03.849172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16790
I20260925 14:54:03.849511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45627
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=36863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:03.851595 16858 log.cc:824] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:03.853981 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:54:03.854296 16858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Bootstrap complete.
I20260925 14:54:03.856525 16858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:54:03.856746 16858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Initialized, Role: FOLLOWER
I20260925 14:54:03.856920 16858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:54:03.856980 16858 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:03.857039 16858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:03.857089 16858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:54:03.858117 16858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:54:03.858211 16858 leader_election.cc:302] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2aeb737939040dfbe2ed5d4304730b8; no voters:
I20260925 14:54:03.858415 16858 leader_election.cc:288] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:54:03.858475 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:54:03.858594 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [term 3 LEADER]: Becoming Leader. State: Replica: d2aeb737939040dfbe2ed5d4304730b8, State: Running, Role: LEADER
I20260925 14:54:03.858681 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } }
I20260925 14:54:03.858685 16858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:03.859004 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:54:03.859179 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:03.859395 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2aeb737939040dfbe2ed5d4304730b8. Latest consensus state: current_term: 3 leader_uuid: "d2aeb737939040dfbe2ed5d4304730b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2aeb737939040dfbe2ed5d4304730b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45493 } } }
I20260925 14:54:03.859484 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:03.859648 16869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:03.860532 16869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:03.860656 16869 catalog_manager.cc:1259] Loaded cluster ID: 11042834a73c4def9fd0c302aac90a44
I20260925 14:54:03.860708 16869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:03.861688 16869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:03.861884 16869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Loaded TSK: 0
I20260925 14:54:03.861940 16869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2aeb737939040dfbe2ed5d4304730b8: Loaded TSK: 1
F20260925 14:54:03.862172 16869 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8c250369ad google::LogMessage::Fail() at ??:0
@ 0x7f8c2503988a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c250364de google::LogMessage::Flush() at ??:0
@ 0x7f8c25037a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c277a3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8c277a0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8c2cf901cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8c2cf9048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8c2cf90594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8c2cf56c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8c2cf60ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8c26f040d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8c26ef9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c27d5d6db start_thread at ??:0
@ 0x7f8c23b3461f clone at ??:0
W20260925 14:54:03.948693 16860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:03.948916 16860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:03.948956 16860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:03.950117 16860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:03.950186 16860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:03.950233 16860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:03.951768 16860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36863
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:03.952039 16860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:03.952270 16860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:03.957355 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:03.957458 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:03.957358 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:03.957662 16860 server_base.cc:1034] running on GCE node
I20260925 14:54:03.957860 16860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:03.958137 16860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:03.959342 16860 hybrid_clock.cc:648] HybridClock initialized: now 1790348043959316 us; error 34 us; skew 500 ppm
Sep 25 14:54:03 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:03.962217 16860 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:54:03.962729 16860 webserver.cc:460] Webserver started at http://127.0.98.1:36863/ using document root <none> and password file <none>
I20260925 14:54:03.962975 16860 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:03.963045 16860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:03.964394 16860 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:03.965169 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:03.965360 16860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:03.965461 16860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "69efcb4590ad6f2f69ed0f175b2fda84"
server_key_iv: "d955da3f4d8c0658d50a5984a824dd12"
server_key_version: "encryptionkey@0"
I20260925 14:54:03.965822 16860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:03.979854 16860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:03.982259 16860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:03.982457 16860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:03.982702 16860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:03.983157 16916 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:54:03.985097 16860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:54:03.985159 16860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:54:03.985208 16860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:54:03.985826 16860 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:54:03.985882 16860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:03.985944 16916 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap starting.
I20260925 14:54:03.993111 16860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45627
I20260925 14:54:03.993258 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45627 every 8 connection(s)
I20260925 14:54:03.993534 16860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:03.995062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 16860
I20260925 14:54:03.995390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44911
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=42343
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:03.997530 16916 log.cc:824] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:03.999874 16916 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:54:04.000173 16916 tablet_bootstrap.cc:492] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Bootstrap complete.
I20260925 14:54:04.001094 16916 ts_tablet_manager.cc:1397] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260925 14:54:04.002147 16916 raft_consensus.cc:348] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:04.002456 16916 raft_consensus.cc:729] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbd1cbfb1cc4e1286638c624ebe64d6, State: Initialized, Role: FOLLOWER
I20260925 14:54:04.002612 16916 consensus_queue.cc:260] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:04.002902 16916 ts_tablet_manager.cc:1428] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260925 14:54:04.082777 17026 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:03.994261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:49531)
0925 14:54:03.994486 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:03.994544 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:03.995163 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:04.082552 (+ 87389us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":141,"threads_started":1}
W20260925 14:54:04.083033 17024 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45493 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45493: Client connection negotiation failed: client connection to 127.0.98.62:45493: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:54:04.083392 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.083278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:50409)
0925 14:54:04.083330 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.083371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:54:04.083714 17026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.083636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:48947)
0925 14:54:04.083678 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.083700 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:54:04.083828 17024 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:54:04.104368 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:04.104593 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:04.104633 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:04.105777 17027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:04.105836 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:04.105880 17027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:04.107450 17027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42343
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:04.107705 17027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:04.107923 17027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:04.113010 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:04.113106 17027 server_base.cc:1034] running on GCE node
W20260925 14:54:04.113018 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:04.113008 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:04.113437 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:04.113631 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:04.114790 17027 hybrid_clock.cc:648] HybridClock initialized: now 1790348044114785 us; error 35 us; skew 500 ppm
Sep 25 14:54:04 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:04.117136 17027 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:54:04.117580 17027 webserver.cc:460] Webserver started at http://127.0.98.2:42343/ using document root <none> and password file <none>
I20260925 14:54:04.117794 17027 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:04.117858 17027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:04.119195 17027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:04.119757 17044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.119916 17027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:04.120014 17027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "ef00f20eef7b418ea4fd5e68c9652a53"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "7f9c4657aa62d5e1c21eb69d41e24c34"
server_key_iv: "70d0887e5f861feca088a94f9a677c62"
server_key_version: "encryptionkey@0"
I20260925 14:54:04.120349 17027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:04.140607 17027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:04.142474 17027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:04.142637 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:04.142872 17027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:04.143208 17027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:04.143266 17027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.143311 17027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:04.143339 17027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.149245 17027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44911
I20260925 14:54:04.149302 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44911 every 8 connection(s)
I20260925 14:54:04.149670 17027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:04.150794 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.150430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:38857)
0925 14:54:04.150580 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.150653 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260925 14:54:04.151211 17158 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45493 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45493: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
I20260925 14:54:04.151350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17027
I20260925 14:54:04.151530 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.151421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:54921)
0925 14:54:04.151478 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.151504 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:54:04.151678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34911
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45493
--webserver_port=45089
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:04.151844 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.151757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:50441)
0925 14:54:04.151801 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.151825 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:54:04.151980 17158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:54:04.231801 17161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:04.231998 17161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:04.232020 17161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:04.233175 17161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:04.233217 17161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:04.233244 17161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:04.234714 17161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45089
--tserver_master_addrs=127.0.98.62:45493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:04.234949 17161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:04.235188 17161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:04.240229 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:04.240266 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:04.240236 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:04.240329 17161 server_base.cc:1034] running on GCE node
I20260925 14:54:04.240628 17161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:04.240872 17161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:04.242030 17161 hybrid_clock.cc:648] HybridClock initialized: now 1790348044242011 us; error 29 us; skew 500 ppm
Sep 25 14:54:04 dist-test-slave-gpc1 krb5kdc[15721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:04.249505 17161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:54:04.249991 17161 webserver.cc:460] Webserver started at http://127.0.98.3:45089/ using document root <none> and password file <none>
I20260925 14:54:04.250239 17161 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:04.250312 17161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:04.251794 17161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:04.252435 17176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.252609 17161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:04.252717 17161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "f0355585ffa2448eb735146e1fd2eafc"
format_stamp: "Formatted at 2026-09-25 14:53:22 on dist-test-slave-gpc1"
server_key: "fa13e6376745e239cbf2cba90893e56d"
server_key_iv: "35d3e3c4d35ec69e52f11fc8900923bf"
server_key_version: "encryptionkey@0"
I20260925 14:54:04.253096 17161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:04.262720 17161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:04.264611 17161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:04.264777 17161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:04.265007 17161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:04.265332 17161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:04.265388 17161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.265424 17161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:04.265455 17161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:04.272001 17161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34911
I20260925 14:54:04.272040 17289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34911 every 8 connection(s)
I20260925 14:54:04.272387 17161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:04.273422 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.273076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:48439)
0925 14:54:04.273222 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.273288 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
W20260925 14:54:04.273837 17290 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45493 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45493: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
I20260925 14:54:04.274111 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.274027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:33647)
0925 14:54:04.274067 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.274093 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:54:04.274457 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:04.274397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:48091)
0925 14:54:04.274422 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:04.274443 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 14:54:04.274580 17290 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:54:04.276371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17161
I20260925 14:54:05.084406 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.084167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:37919)
0925 14:54:05.084291 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.084331 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 14:54:05.152478 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.152280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:49763)
0925 14:54:05.152391 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.152426 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260925 14:54:05.275118 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.274916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:60515)
0925 14:54:05.275032 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.275071 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 14:54:05.389288 17297 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:05.389420 17297 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:05.389839 17297 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
Sep 25 14:54:05 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 14:54:05 dist-test-slave-gpc1 krb5kdc[15721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 14:54:05.400552 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.390224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33037 (local address 127.0.98.2:44911)
0925 14:54:05.390376 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.390383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.390411 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:54:05.390934 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.390937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.390997 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.391154 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:05.392020 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.392932 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.393916 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.394315 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.394485 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.396877 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:05.396880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:05.396880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:05.396889 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:05.398844 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399357 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:05.399366 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:05.399367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:05.399445 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399708 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:05.399709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:05.399710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:05.399942 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:05.400036 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.400312 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.400435 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260925 14:54:05.400549 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.390022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34911 (local address 127.0.98.1:46677)
0925 14:54:05.390073 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.390086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:05.390160 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:05.391227 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:05.391230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:05.391674 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:05.391875 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.391880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.393045 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.393046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.393824 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.393825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.394448 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.394449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.394573 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.395158 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:05.395167 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:05.397128 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:05.399029 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:05.399030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:05.399257 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:05.399516 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:05.399517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:05.399639 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:05.400026 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:05.400032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:05.400150 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:54:05.400256 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:54:05.400447 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":5}
I20260925 14:54:05.400573 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.390143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46677 (local address 127.0.98.3:34911)
0925 14:54:05.390282 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:05.390295 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:05.390332 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:54:05.390934 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:05.390937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:05.390997 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:05.391350 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:05.392020 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.392934 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.393912 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.394314 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.394476 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.397230 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:05.397233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:05.397234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:05.397247 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:05.398941 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399334 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:05.399338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:05.399338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:05.399436 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399707 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:05.399708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:05.399709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:05.399942 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:05.400031 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:05.400313 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:54:05.400437 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260925 14:54:05.400755 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:05.390069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44911 (local address 127.0.98.1:33037)
0925 14:54:05.390227 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:05.390247 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:05.390302 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:05.391228 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:05.391231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:05.391667 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:05.391875 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.391880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.393039 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.393040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.393823 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:05.393826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:05.394460 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:05.394461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:05.394574 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:05.395207 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:05.395212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:05.396761 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:05.398925 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:05.398932 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:05.398934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:05.399259 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:05.399541 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:05.399541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:05.399542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:05.399642 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:05.400023 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:05.400029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:05.400149 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:54:05.400257 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:54:05.400529 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":197,"thread_start_us":94,"threads_started":1}
W20260925 14:54:05.401965 16911 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
W20260925 14:54:05.402050 16913 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
I20260925 14:54:05.402143 16913 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6; no voters: ef00f20eef7b418ea4fd5e68c9652a53, f0355585ffa2448eb735146e1fd2eafc
I20260925 14:54:05.402264 17297 raft_consensus.cc:2726] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:54:06.085130 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:06.084906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:42219)
0925 14:54:06.085042 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:06.085083 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260925 14:54:06.153153 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:06.152952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:40057)
0925 14:54:06.153080 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:06.153115 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260925 14:54:06.275772 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:06.275564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:53239)
0925 14:54:06.275703 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:06.275737 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260925 14:54:07.075884 17304 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:07.075976 17304 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:07.076133 17304 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
W20260925 14:54:07.076520 16913 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
W20260925 14:54:07.076645 16911 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
I20260925 14:54:07.076707 16911 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6; no voters: ef00f20eef7b418ea4fd5e68c9652a53, f0355585ffa2448eb735146e1fd2eafc
I20260925 14:54:07.076814 17304 raft_consensus.cc:2726] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:54:07.085906 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.085691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:40239)
0925 14:54:07.085819 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.085851 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260925 14:54:07.153939 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.153717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:46987)
0925 14:54:07.153855 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.153897 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260925 14:54:07.276556 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:07.276311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:46571)
0925 14:54:07.276461 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:07.276503 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260925 14:54:08.086725 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.086512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:43643)
0925 14:54:08.086639 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.086679 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260925 14:54:08.154690 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.154476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:37157)
0925 14:54:08.154610 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.154645 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260925 14:54:08.277333 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:08.277121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:35235)
0925 14:54:08.277254 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:08.277292 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 14:54:09.087481 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:09.087274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:50091)
0925 14:54:09.087402 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:09.087434 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260925 14:54:09.155404 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:09.155183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:39175)
0925 14:54:09.155318 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:09.155360 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260925 14:54:09.277951 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:09.277759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:36263)
0925 14:54:09.277871 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:09.277911 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260925 14:54:09.658028 17314 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:09.658133 17314 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:09.658370 17314 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
W20260925 14:54:09.658756 16911 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
W20260925 14:54:09.658830 16913 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
I20260925 14:54:09.658895 16913 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6; no voters: ef00f20eef7b418ea4fd5e68c9652a53, f0355585ffa2448eb735146e1fd2eafc
I20260925 14:54:09.658977 17314 raft_consensus.cc:2726] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:54:10.088299 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:10.088059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:35627)
0925 14:54:10.088214 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:10.088252 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
I20260925 14:54:10.156044 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:10.155836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:50483)
0925 14:54:10.155958 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:10.156001 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260925 14:54:10.278728 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:10.278515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:55449)
0925 14:54:10.278652 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:10.278687 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260925 14:54:11.089064 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.088835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:36995)
0925 14:54:11.088968 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.089005 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260925 14:54:11.156764 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.156463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:47225)
0925 14:54:11.156661 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.156701 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260925 14:54:11.279422 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:11.279223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:35421)
0925 14:54:11.279341 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:11.279379 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 14:54:11.865218 17321 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:11.865330 17321 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:11.865509 17321 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
W20260925 14:54:11.865936 16911 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
W20260925 14:54:11.866020 16913 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
I20260925 14:54:11.866078 16913 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6; no voters: ef00f20eef7b418ea4fd5e68c9652a53, f0355585ffa2448eb735146e1fd2eafc
I20260925 14:54:11.866147 17321 raft_consensus.cc:2726] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:54:12.089788 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.089571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:50597)
0925 14:54:12.089701 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.089741 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260925 14:54:12.157421 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.157189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:48899)
0925 14:54:12.157312 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.157354 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260925 14:54:12.280117 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:12.279882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:51573)
0925 14:54:12.280003 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:12.280042 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260925 14:54:13.090539 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.090285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:39557)
0925 14:54:13.090452 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.090492 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260925 14:54:13.158054 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.157845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:47017)
0925 14:54:13.157967 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.158006 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
I20260925 14:54:13.280810 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:13.280619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:34597)
0925 14:54:13.280733 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:13.280763 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 14:54:14.091248 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.091036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:49441)
0925 14:54:14.091164 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.091203 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 14:54:14.158761 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.158550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:48411)
0925 14:54:14.158663 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.158705 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260925 14:54:14.281488 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:14.281267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:41849)
0925 14:54:14.281402 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:14.281442 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260925 14:54:15.092159 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.091910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:53881)
0925 14:54:15.092055 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.092111 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260925 14:54:15.159374 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.159154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:59461)
0925 14:54:15.159287 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.159328 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260925 14:54:15.282228 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:15.282003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:42937)
0925 14:54:15.282146 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:15.282184 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260925 14:54:16.092919 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.092694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:56253)
0925 14:54:16.092832 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.092873 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260925 14:54:16.160024 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.159813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:53561)
0925 14:54:16.159938 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.159980 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260925 14:54:16.282879 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:16.282672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:47631)
0925 14:54:16.282795 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:16.282835 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260925 14:54:16.932716 17337 raft_consensus.cc:482] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:16.932806 17337 raft_consensus.cc:504] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0355585ffa2448eb735146e1fd2eafc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34911 } } peers { permanent_uuid: "ef00f20eef7b418ea4fd5e68c9652a53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44911 } } peers { permanent_uuid: "4cbd1cbfb1cc4e1286638c624ebe64d6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45627 } }
I20260925 14:54:16.932960 17337 leader_election.cc:288] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911), ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911)
W20260925 14:54:16.933287 16911 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0355585ffa2448eb735146e1fd2eafc (127.0.98.3:34911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
W20260925 14:54:16.933362 16913 leader_election.cc:341] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ef00f20eef7b418ea4fd5e68c9652a53 (127.0.98.2:44911): Not found: Tablet not found: 24927187f30e449894c23df664d098b4
I20260925 14:54:16.933470 16913 leader_election.cc:302] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4cbd1cbfb1cc4e1286638c624ebe64d6; no voters: ef00f20eef7b418ea4fd5e68c9652a53, f0355585ffa2448eb735146e1fd2eafc
I20260925 14:54:16.933550 17337 raft_consensus.cc:2726] T 24927187f30e449894c23df664d098b4 P 4cbd1cbfb1cc4e1286638c624ebe64d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:54:17.093641 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.093433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:58197)
0925 14:54:17.093557 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.093594 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260925 14:54:17.160691 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.160502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:36543)
0925 14:54:17.160612 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.160647 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
I20260925 14:54:17.283614 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:17.283387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:38337)
0925 14:54:17.283526 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:17.283566 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 14:54:18.094297 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.094094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:33231)
0925 14:54:18.094216 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.094253 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260925 14:54:18.161335 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.161112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:52605)
0925 14:54:18.161247 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.161290 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 14:54:18.284382 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:18.284177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:50117)
0925 14:54:18.284300 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:18.284337 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 14:54:19.095156 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.094930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.1:35649)
0925 14:54:19.095067 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.095109 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260925 14:54:19.162022 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.161824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.2:36371)
0925 14:54:19.161940 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.161973 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260925 14:54:19.284135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 16860
I20260925 14:54:19.285399 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:19.284923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45493 (local address 127.0.98.3:46693)
0925 14:54:19.285327 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:19.285355 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":55,"threads_started":1}
I20260925 14:54:19.288836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17027
I20260925 14:54:19.293035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17161
2026-09-25T14:54:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58406 ms)
[----------] 30 tests from SecurityITest (234238 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:19 dist-test-slave-gpc1 krb5kdc[17349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:19 dist-test-slave-gpc1 krb5kdc[17349](info): set up 2 sockets
Sep 25 14:54:19 dist-test-slave-gpc1 krb5kdc[17349](info): commencing operation
krb5kdc: starting...
W20260925 14:54:20.342085 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:20.454427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:20.536895 17362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:20.537141 17362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:20.537178 17362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:20.538329 17362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:20.538445 17362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:20.538475 17362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:20.538513 17362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:20.538537 17362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:20.540019 17362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:20.540302 17362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:20.540540 17362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:20.545367 17368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:20.545421 17367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:20.545359 17370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:20.545624 17362 server_base.cc:1034] running on GCE node
I20260925 14:54:20.545861 17362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:20.546152 17362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:20.547300 17362 hybrid_clock.cc:648] HybridClock initialized: now 1790348060547261 us; error 52 us; skew 500 ppm
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:20.549836 17362 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:20.550247 17362 webserver.cc:460] Webserver started at http://127.0.98.62:33633/ using document root <none> and password file <none>
I20260925 14:54:20.550504 17362 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:20.550550 17362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:20.550694 17362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:20.551751 17362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "302eeabf3a304344913a984abea28aa3"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "7421a6fed9433684efc401a70248ff44"
server_key_iv: "e8c47cda698ee69ac8375894cc709696"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.552105 17362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "302eeabf3a304344913a984abea28aa3"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "7421a6fed9433684efc401a70248ff44"
server_key_iv: "e8c47cda698ee69ac8375894cc709696"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.553447 17362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:20.554145 17377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.554322 17362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:20.554478 17362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "302eeabf3a304344913a984abea28aa3"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "7421a6fed9433684efc401a70248ff44"
server_key_iv: "e8c47cda698ee69ac8375894cc709696"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.554560 17362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:20.571734 17362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:20.574097 17362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:20.574259 17362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:20.578671 17362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38897
I20260925 14:54:20.578707 17429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38897 every 8 connection(s)
I20260925 14:54:20.579144 17362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:20.580102 17430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:20.582367 17430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: Bootstrap starting.
I20260925 14:54:20.582938 17430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:20.583190 17430 log.cc:824] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:20.583856 17430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: No bootstrap required, opened a new log
I20260925 14:54:20.585343 17430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } }
I20260925 14:54:20.585494 17430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:20.585518 17430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 302eeabf3a304344913a984abea28aa3, State: Initialized, Role: FOLLOWER
I20260925 14:54:20.585621 17430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } }
I20260925 14:54:20.585700 17430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:20.585732 17430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:20.585783 17430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:20.586308 17430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } }
I20260925 14:54:20.586445 17430 leader_election.cc:302] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 302eeabf3a304344913a984abea28aa3; no voters:
I20260925 14:54:20.586632 17430 leader_election.cc:288] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:20.586674 17433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:20.586851 17433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 302eeabf3a304344913a984abea28aa3, State: Running, Role: LEADER
I20260925 14:54:20.586903 17430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:20.586984 17433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } }
I20260925 14:54:20.587545 17434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "302eeabf3a304344913a984abea28aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } } }
I20260925 14:54:20.587639 17434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:20.588004 17442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:20.588121 17435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 302eeabf3a304344913a984abea28aa3. Latest consensus state: current_term: 1 leader_uuid: "302eeabf3a304344913a984abea28aa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "302eeabf3a304344913a984abea28aa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38897 } } }
I20260925 14:54:20.588217 17435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:20.588586 17442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:20.588999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17362
I20260925 14:54:20.589128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:20.589390 392 external_mini_cluster.cc:1428] Setting key 5e0b8cd4f3691caec5ee2b8d2862d56e
I20260925 14:54:20.590584 17442 catalog_manager.cc:1347] Generated new cluster ID: a4882bb0ed924d82abd183707147c653
I20260925 14:54:20.590662 17442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:20.602308 17442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:20.602846 17442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:20.603653 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.589852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.62:38897)
0925 14:54:20.590102 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.590111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.590142 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:54:20.590254 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.590261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.590317 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.594550 (+ 4233us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.595219 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.596114 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.596878 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.597321 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.597430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.599861 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:20.599864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:20.599865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:20.599876 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:20.601579 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.602008 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.602011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.602011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.602142 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.602480 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.602481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.602481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.602773 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:20.602956 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:20.603063 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:54:20.603402 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260925 14:54:20.613204 17442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: Generated new TSK 0
I20260925 14:54:20.613481 17442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:54:20.645756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:20.741451 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:20.741667 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:20.741727 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:20.742977 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:20.743023 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:20.743050 17456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:20.744496 17456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:20.744683 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:20.744853 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:20.749876 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:20.749886 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:20.749874 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:20.750131 17456 server_base.cc:1034] running on GCE node
I20260925 14:54:20.750397 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:20.750633 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:20.751777 17456 hybrid_clock.cc:648] HybridClock initialized: now 1790348060751772 us; error 35 us; skew 500 ppm
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:20.754278 17456 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:54:20.754784 17456 webserver.cc:460] Webserver started at http://127.0.98.1:38089/ using document root <none> and password file <none>
I20260925 14:54:20.754974 17456 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:20.755015 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:20.755102 17456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:20.756098 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "4fddb677d2a8ea9770d12e6afa2d2937"
server_key_iv: "28515e2d63a758f669e1d22cbbba4c6f"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.756403 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "4fddb677d2a8ea9770d12e6afa2d2937"
server_key_iv: "28515e2d63a758f669e1d22cbbba4c6f"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.757709 17456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:54:20.758445 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.758734 17456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:20.758831 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "4fddb677d2a8ea9770d12e6afa2d2937"
server_key_iv: "28515e2d63a758f669e1d22cbbba4c6f"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.758908 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:20.795280 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:20.797237 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:20.797421 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:20.797664 17456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:20.797986 17456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:20.798035 17456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.798075 17456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:20.798103 17456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.804262 17456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37093
I20260925 14:54:20.804312 17584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37093 every 8 connection(s)
I20260925 14:54:20.804643 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:20.811612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17456
I20260925 14:54:20.811719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:20.811928 392 external_mini_cluster.cc:1428] Setting key 65f79c5df882c0bd5afb0440d007031d
I20260925 14:54:20.814450 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.805376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53915 (local address 127.0.98.62:38897)
0925 14:54:20.805460 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.805463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.805645 (+ 182us) server_negotiation.cc:407] Connection header received
0925 14:54:20.806698 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.806700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.806713 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.806765 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.807416 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.807928 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.808947 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.809173 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.809257 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.811375 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:20.811377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:20.811377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:20.811384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:20.812878 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.813355 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.813356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.813357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.813409 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.813681 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.813681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.813681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.813781 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:20.813831 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:20.814180 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 14:54:20.814284 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:20.814563 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.805328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38897 (local address 127.0.98.1:53915)
0925 14:54:20.805511 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:20.805561 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:20.806580 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:20.806842 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:20.806852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:20.807116 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:20.807311 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.807315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.808019 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.808020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.808840 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.808842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.809266 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.809267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.809405 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.809857 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:20.809866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:20.811254 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:20.812951 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:20.812957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:20.812959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:20.813269 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:20.813475 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:20.813476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:20.813476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:20.813618 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:20.813841 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:20.813847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:20.814017 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 14:54:20.814140 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 14:54:20.814300 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260925 14:54:20.815078 17585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38897
I20260925 14:54:20.815183 17585 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:20.815384 17585 heartbeater.cc:511] Master 127.0.98.62:38897 requested a full tablet report, sending...
I20260925 14:54:20.815821 17394 ts_manager.cc:194] Registered new tserver with Master: 8d30c7f7db9c4cb3b27d7f47e4db6d8f (127.0.98.1:37093)
I20260925 14:54:20.816504 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53915
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:54:20.837188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:20.919636 17590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:20.919875 17590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:20.919919 17590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:20.921088 17590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:20.921159 17590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:20.921207 17590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:20.922822 17590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:20.923100 17590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:20.923336 17590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:20.928766 17596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:20.928856 17590 server_base.cc:1034] running on GCE node
W20260925 14:54:20.928757 17595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:20.928774 17598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:20.929255 17590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:20.929530 17590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:20.930671 17590 hybrid_clock.cc:648] HybridClock initialized: now 1790348060930662 us; error 27 us; skew 500 ppm
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:20.933189 17590 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:54:20.933691 17590 webserver.cc:460] Webserver started at http://127.0.98.2:44279/ using document root <none> and password file <none>
I20260925 14:54:20.933914 17590 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:20.933960 17590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:20.934137 17590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:20.935338 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "7596f711167b434087afcb75376fddc1"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "9aaf401b5366c84c2541be65cf798123"
server_key_iv: "d5cffccb2c92a2384c3f5c7ea770cfcc"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.935703 17590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7596f711167b434087afcb75376fddc1"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "9aaf401b5366c84c2541be65cf798123"
server_key_iv: "d5cffccb2c92a2384c3f5c7ea770cfcc"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.937042 17590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:20.937785 17605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.937970 17590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:20.938059 17590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "7596f711167b434087afcb75376fddc1"
format_stamp: "Formatted at 2026-09-25 14:54:20 on dist-test-slave-gpc1"
server_key: "9aaf401b5366c84c2541be65cf798123"
server_key_iv: "d5cffccb2c92a2384c3f5c7ea770cfcc"
server_key_version: "encryptionkey@0"
I20260925 14:54:20.938148 17590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:20.947180 17590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:20.949426 17590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:20.949577 17590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:20.949815 17590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:20.950145 17590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:20.950178 17590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.950201 17590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:20.950232 17590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:20.956387 17590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43221
I20260925 14:54:20.956486 17718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43221 every 8 connection(s)
I20260925 14:54:20.956815 17590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:20.962229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17590
I20260925 14:54:20.962384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:20.962627 392 external_mini_cluster.cc:1428] Setting key b0856a31794ce2660f6b944fe553ab09
Sep 25 14:54:20 dist-test-slave-gpc1 krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:20.967180 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.957660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58511 (local address 127.0.98.62:38897)
0925 14:54:20.957799 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:20.957801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:20.957949 (+ 148us) server_negotiation.cc:407] Connection header received
0925 14:54:20.959025 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:20.959026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:20.959039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:20.959116 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:20.959747 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.960290 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.961259 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.961449 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.961505 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.964046 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:20.964048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:20.964048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:20.964057 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:20.965574 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.966098 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.966099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.966099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.966151 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:20.966453 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:20.966454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:20.966454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:20.966570 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:20.966623 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:20.966965 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 14:54:20.967043 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 14:54:20.967444 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:20.957624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38897 (local address 127.0.98.2:58511)
0925 14:54:20.957810 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:20.957854 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:20.958919 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:20.959176 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:20.959180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:20.959441 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:20.959638 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.959641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.960375 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.960375 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.961185 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:20.961187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:20.961558 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:20.961559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:20.961696 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:20.962206 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:20.962218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:20.963926 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:20.965664 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:20.965670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:20.965673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:20.966018 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:20.966213 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:20.966214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:20.966214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:20.966388 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:20.966636 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:20.966643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:20.966823 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:54:20.966934 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:54:20.967149 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
I20260925 14:54:20.968034 17719 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38897
I20260925 14:54:20.968134 17719 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:20.968341 17719 heartbeater.cc:511] Master 127.0.98.62:38897 requested a full tablet report, sending...
I20260925 14:54:20.968760 17394 ts_manager.cc:194] Registered new tserver with Master: 7596f711167b434087afcb75376fddc1 (127.0.98.2:43221)
I20260925 14:54:20.969156 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58511
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:54:20.989022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:21.069494 17724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:21.069697 17724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:21.069720 17724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:21.070840 17724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:21.070873 17724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:21.070899 17724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:21.072324 17724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:21.072513 17724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:21.072685 17724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:21.077649 17729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.077637 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.077637 17732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:21.077893 17724 server_base.cc:1034] running on GCE node
I20260925 14:54:21.078248 17724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:21.078560 17724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:21.079730 17724 hybrid_clock.cc:648] HybridClock initialized: now 1790348061079707 us; error 38 us; skew 500 ppm
Sep 25 14:54:21 dist-test-slave-gpc1 krb5kdc[17349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:21.082221 17724 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:54:21.082690 17724 webserver.cc:460] Webserver started at http://127.0.98.3:45253/ using document root <none> and password file <none>
I20260925 14:54:21.082927 17724 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:21.082973 17724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:21.083148 17724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:21.084208 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dd27596e14642608403942dc8260a88"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "0538efd7e40aa131d90fe6bc55041e0a"
server_key_iv: "5edcce264db167591678a60c7178cea5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.084566 17724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dd27596e14642608403942dc8260a88"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "0538efd7e40aa131d90fe6bc55041e0a"
server_key_iv: "5edcce264db167591678a60c7178cea5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.085896 17724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:21.086691 17739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.086865 17724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:21.086949 17724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "6dd27596e14642608403942dc8260a88"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "0538efd7e40aa131d90fe6bc55041e0a"
server_key_iv: "5edcce264db167591678a60c7178cea5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.087028 17724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:21.094837 17724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:21.096562 17724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:21.096673 17724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:21.096841 17724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:21.097137 17724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:21.097170 17724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.097193 17724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:21.097241 17724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.102981 17724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37291
I20260925 14:54:21.103085 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37291 every 8 connection(s)
I20260925 14:54:21.103420 17724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:21.103771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17724
I20260925 14:54:21.103865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:21.104048 392 external_mini_cluster.cc:1428] Setting key 2f12c5fdce208b1bf325cc967f2e3420
Sep 25 14:54:21 dist-test-slave-gpc1 krb5kdc[17349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:21.113420 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.104376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40661 (local address 127.0.98.62:38897)
0925 14:54:21.104432 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.104434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.104792 (+ 358us) server_negotiation.cc:407] Connection header received
0925 14:54:21.105792 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.105793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.105803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.105868 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.106523 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.107041 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.108009 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.108187 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.108237 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.110577 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.110578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.110579 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:21.110593 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.111884 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:21.112308 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:21.112309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:21.112310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:21.112375 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:21.112644 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:21.112645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:21.112645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:21.112803 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.112904 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.113198 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.113320 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:54:21.113579 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.104393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38897 (local address 127.0.98.3:40661)
0925 14:54:21.104669 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.104721 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.105698 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.105908 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.105912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.106181 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.106441 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.106447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.107118 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.107119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.107934 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.107935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.108256 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.108257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.108359 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.108920 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:21.108928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.110460 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:21.111973 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:21.111981 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:21.111983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:21.112215 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:21.112445 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:21.112446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:21.112446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:21.112565 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:21.112903 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:21.112911 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:21.113055 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.113160 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.113335 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260925 14:54:21.114045 17853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38897
I20260925 14:54:21.114131 17853 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:21.114272 17853 heartbeater.cc:511] Master 127.0.98.62:38897 requested a full tablet report, sending...
I20260925 14:54:21.114652 17394 ts_manager.cc:194] Registered new tserver with Master: 6dd27596e14642608403942dc8260a88 (127.0.98.3:37291)
I20260925 14:54:21.115033 17394 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40661
I20260925 14:54:21.115734 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:21.122844 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.117016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.0.98.62:38897)
0925 14:54:21.117071 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.117073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.117185 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:54:21.117263 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.117264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.117274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.117342 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.117904 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.118425 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.119166 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.119353 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.119426 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.120533 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.120535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.120535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:21.120542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.121622 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:21.121972 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:21.121973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:21.121974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:21.122028 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:21.122242 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:21.122243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:21.122244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:21.122453 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.122560 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.122669 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.122747 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:21.124291 17394 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:21.126072 17394 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:21.132884 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37291 (local address 127.0.0.1:45474)
0925 14:54:21.128584 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.128613 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.128704 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.129200 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.129204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.129205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.129279 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.129281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130609 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.130610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.131731 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.131733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132646 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.132702 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132705 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.132759 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.132789 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260925 14:54:21.132800 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37093 (local address 127.0.0.1:55278)
0925 14:54:21.128972 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.128982 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.129047 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.129319 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.129320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.129322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.129386 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.129388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130385 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.130386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.131766 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.131768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132451 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.132533 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132547 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.132647 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.132705 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":45,"thread_start_us":98,"threads_started":1}
I20260925 14:54:21.132994 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.0.98.1:37093)
0925 14:54:21.128978 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.128983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.129033 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:54:21.129116 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.129118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.129130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.129224 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.129467 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130244 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.131852 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132354 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132439 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132503 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.132843 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.132900 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260925 14:54:21.132997 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43221 (local address 127.0.0.1:60486)
0925 14:54:21.129299 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.129309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.129357 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.129561 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.129561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.129563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.129603 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.129605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130800 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.130801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.131743 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.131746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132726 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.132809 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132814 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.132865 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.132905 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":20,"thread_start_us":64,"threads_started":1}
I20260925 14:54:21.133455 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.0.98.2:43221)
0925 14:54:21.128992 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.128996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.129413 (+ 417us) server_negotiation.cc:407] Connection header received
0925 14:54:21.129458 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.129461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.129474 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.129549 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.129666 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130708 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.131856 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132560 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132631 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.133331 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.133380 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260925 14:54:21.133790 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.128609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.0.98.3:37291)
0925 14:54:21.129026 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.129031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.129042 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:21.129083 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.129085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.129103 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.129199 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.129366 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.130512 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.131827 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.132560 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.132633 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.132689 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.132784 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.132833 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":91,"threads_started":1}
I20260925 14:54:21.133800 17519 tablet_service.cc:1467] Processing CreateTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 (DEFAULT_TABLE table=test-table [id=1ac5d4e257db44f7802ffa5d7603d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.134222 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f257ca20ed81417a8ae8a5a968bf0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.134137 17653 tablet_service.cc:1467] Processing CreateTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 (DEFAULT_TABLE table=test-table [id=1ac5d4e257db44f7802ffa5d7603d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.134497 17653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f257ca20ed81417a8ae8a5a968bf0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.134693 17787 tablet_service.cc:1467] Processing CreateTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 (DEFAULT_TABLE table=test-table [id=1ac5d4e257db44f7802ffa5d7603d20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.135010 17787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f257ca20ed81417a8ae8a5a968bf0162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.136972 17870 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Bootstrap starting.
I20260925 14:54:21.137514 17870 tablet_bootstrap.cc:654] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.138027 17871 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Bootstrap starting.
I20260925 14:54:21.138057 17870 log.cc:824] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.138455 17872 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Bootstrap starting.
I20260925 14:54:21.138746 17871 tablet_bootstrap.cc:654] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.138866 17870 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: No bootstrap required, opened a new log
I20260925 14:54:21.138947 17870 ts_tablet_manager.cc:1397] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:21.139125 17872 tablet_bootstrap.cc:654] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.139317 17871 log.cc:824] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.139740 17872 log.cc:824] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.140092 17871 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: No bootstrap required, opened a new log
I20260925 14:54:21.140198 17871 ts_tablet_manager.cc:1397] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:54:21.140434 17872 tablet_bootstrap.cc:492] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: No bootstrap required, opened a new log
I20260925 14:54:21.140518 17872 ts_tablet_manager.cc:1397] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:21.141005 17870 raft_consensus.cc:348] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.141224 17870 raft_consensus.cc:374] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.141279 17870 raft_consensus.cc:729] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d30c7f7db9c4cb3b27d7f47e4db6d8f, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.141458 17870 consensus_queue.cc:260] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.141868 17870 ts_tablet_manager.cc:1428] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:21.141908 17585 heartbeater.cc:503] Master 127.0.98.62:38897 was elected leader, sending a full tablet report...
I20260925 14:54:21.142320 17871 raft_consensus.cc:348] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.142557 17871 raft_consensus.cc:374] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.142632 17871 raft_consensus.cc:729] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7596f711167b434087afcb75376fddc1, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.142758 17871 consensus_queue.cc:260] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.143045 17871 ts_tablet_manager.cc:1428] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:54:21.143184 17719 heartbeater.cc:503] Master 127.0.98.62:38897 was elected leader, sending a full tablet report...
I20260925 14:54:21.143244 17872 raft_consensus.cc:348] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.143437 17872 raft_consensus.cc:374] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.143479 17872 raft_consensus.cc:729] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dd27596e14642608403942dc8260a88, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.143592 17872 consensus_queue.cc:260] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.143872 17872 ts_tablet_manager.cc:1428] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:21.143904 17853 heartbeater.cc:503] Master 127.0.98.62:38897 was elected leader, sending a full tablet report...
W20260925 14:54:21.207448 17720 tablet.cc:2369] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:21.305299 17586 tablet.cc:2369] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:21.309491 17878 raft_consensus.cc:482] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:21.309630 17878 raft_consensus.cc:504] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.310122 17878 leader_election.cc:288] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d30c7f7db9c4cb3b27d7f47e4db6d8f (127.0.98.1:37093), 7596f711167b434087afcb75376fddc1 (127.0.98.2:43221)
I20260925 14:54:21.314070 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.310625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42703 (local address 127.0.98.2:43221)
0925 14:54:21.310693 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.310695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.310831 (+ 136us) server_negotiation.cc:407] Connection header received
0925 14:54:21.310900 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.310901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.310915 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.310995 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.311180 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.311840 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.313283 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.313648 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.313707 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.313716 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.313951 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.313999 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:54:21.314085 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.310534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43221 (local address 127.0.98.3:42703)
0925 14:54:21.310735 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.310744 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.310836 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.310998 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.310999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.311001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.311086 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.311088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.311928 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.311929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.313179 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.313181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.313762 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.313763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.313829 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.313834 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.313942 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.313987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":146,"threads_started":1}
I20260925 14:54:21.314505 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f257ca20ed81417a8ae8a5a968bf0162" candidate_uuid: "6dd27596e14642608403942dc8260a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7596f711167b434087afcb75376fddc1" is_pre_election: true
I20260925 14:54:21.314563 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.310403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37093 (local address 127.0.98.3:48301)
0925 14:54:21.310485 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.310501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.310590 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.310746 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.310747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.310749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.310821 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.310823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.311757 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.311758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.313178 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.313181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.314258 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.314260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.314367 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.314372 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.314444 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.314487 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:54:21.314589 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.310436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48301 (local address 127.0.98.1:37093)
0925 14:54:21.310504 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.310506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.310588 (+ 82us) server_negotiation.cc:407] Connection header received
0925 14:54:21.310659 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.310660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.310674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.310729 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.310898 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.311650 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.313747 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.314127 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.314219 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.314230 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.314469 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.314514 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:21.314697 17673 raft_consensus.cc:2454] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd27596e14642608403942dc8260a88 in term 0.
I20260925 14:54:21.314937 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f257ca20ed81417a8ae8a5a968bf0162" candidate_uuid: "6dd27596e14642608403942dc8260a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" is_pre_election: true
I20260925 14:54:21.314981 17740 leader_election.cc:302] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd27596e14642608403942dc8260a88, 7596f711167b434087afcb75376fddc1; no voters:
I20260925 14:54:21.315102 17539 raft_consensus.cc:2454] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dd27596e14642608403942dc8260a88 in term 0.
I20260925 14:54:21.315140 17878 raft_consensus.cc:2781] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:21.315261 17878 raft_consensus.cc:482] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:21.315296 17878 raft_consensus.cc:3037] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:21.316040 17878 raft_consensus.cc:504] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.316182 17878 leader_election.cc:288] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [CANDIDATE]: Term 1 election: Requested vote from peers 8d30c7f7db9c4cb3b27d7f47e4db6d8f (127.0.98.1:37093), 7596f711167b434087afcb75376fddc1 (127.0.98.2:43221)
I20260925 14:54:21.316368 17539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f257ca20ed81417a8ae8a5a968bf0162" candidate_uuid: "6dd27596e14642608403942dc8260a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f"
I20260925 14:54:21.316391 17673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f257ca20ed81417a8ae8a5a968bf0162" candidate_uuid: "6dd27596e14642608403942dc8260a88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7596f711167b434087afcb75376fddc1"
I20260925 14:54:21.316461 17539 raft_consensus.cc:3037] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:21.316483 17673 raft_consensus.cc:3037] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:21.317158 17673 raft_consensus.cc:2454] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd27596e14642608403942dc8260a88 in term 1.
I20260925 14:54:21.317314 17539 raft_consensus.cc:2454] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dd27596e14642608403942dc8260a88 in term 1.
I20260925 14:54:21.317365 17740 leader_election.cc:302] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dd27596e14642608403942dc8260a88, 7596f711167b434087afcb75376fddc1; no voters:
I20260925 14:54:21.317469 17878 raft_consensus.cc:2781] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:21.317655 17878 raft_consensus.cc:686] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 1 LEADER]: Becoming Leader. State: Replica: 6dd27596e14642608403942dc8260a88, State: Running, Role: LEADER
I20260925 14:54:21.317759 17878 consensus_queue.cc:237] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } }
I20260925 14:54:21.318636 17393 catalog_manager.cc:5495] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dd27596e14642608403942dc8260a88 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6dd27596e14642608403942dc8260a88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dd27596e14642608403942dc8260a88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 } health_report { overall_health: UNKNOWN } } }
W20260925 14:54:21.354012 17854 tablet.cc:2369] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:21.361109 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.358620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.0.98.3:37291)
0925 14:54:21.358670 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.358673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.358701 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:54:21.358744 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.358745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.358764 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.358829 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:21.358954 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.359518 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.360319 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.360529 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.360587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.360731 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:21.360837 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:21.360904 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.360944 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.360963 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:21.363222 17673 raft_consensus.cc:1261] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 1 FOLLOWER]: Refusing update from remote peer 6dd27596e14642608403942dc8260a88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:21.363222 17539 raft_consensus.cc:1261] T f257ca20ed81417a8ae8a5a968bf0162 P 8d30c7f7db9c4cb3b27d7f47e4db6d8f [term 1 FOLLOWER]: Refusing update from remote peer 6dd27596e14642608403942dc8260a88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:21.363700 17880 consensus_queue.cc:1035] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7596f711167b434087afcb75376fddc1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:21.363983 17878 consensus_queue.cc:1035] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30c7f7db9c4cb3b27d7f47e4db6d8f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:21.366638 17885 mvcc.cc:205] Tried to move back new op lower bound from 7333265659340419072 to 7333265659158155264. Current Snapshot: MvccSnapshot[applied={T|T < 7333265659340419072}]
I20260925 14:54:21.367790 17884 mvcc.cc:205] Tried to move back new op lower bound from 7333265659340419072 to 7333265659158155264. Current Snapshot: MvccSnapshot[applied={T|T < 7333265659340419072 or (T in {7333265659340419072})}]
I20260925 14:54:21.368702 17393 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:21.368815 17393 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:21.369655 17393 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 302eeabf3a304344913a984abea28aa3: Sending DeleteTablet for 3 replicas of tablet f257ca20ed81417a8ae8a5a968bf0162
I20260925 14:54:21.370085 17519 tablet_service.cc:1514] Processing DeleteTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55278
I20260925 14:54:21.370102 17787 tablet_service.cc:1514] Processing DeleteTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45474
I20260925 14:54:21.370090 17653 tablet_service.cc:1514] Processing DeleteTablet for tablet f257ca20ed81417a8ae8a5a968bf0162 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60486
I20260925 14:54:21.370488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17456
I20260925 14:54:21.372153 17894 tablet_replica.cc:330] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: stopping tablet replica
I20260925 14:54:21.372264 17894 raft_consensus.cc:2229] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:21.372377 17894 raft_consensus.cc:2258] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:21.372527 17895 tablet_replica.cc:330] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: stopping tablet replica
I20260925 14:54:21.372689 17895 raft_consensus.cc:2229] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:21.372804 17894 ts_tablet_manager.cc:1905] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:21.372841 17895 raft_consensus.cc:2258] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:21.373320 17895 ts_tablet_manager.cc:1905] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:21.374749 17894 ts_tablet_manager.cc:1918] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:21.374890 17894 log.cc:1196] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/f257ca20ed81417a8ae8a5a968bf0162
I20260925 14:54:21.375279 17894 ts_tablet_manager.cc:1939] T f257ca20ed81417a8ae8a5a968bf0162 P 6dd27596e14642608403942dc8260a88: Deleting consensus metadata
I20260925 14:54:21.375396 17895 ts_tablet_manager.cc:1918] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:21.375528 17895 log.cc:1196] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/f257ca20ed81417a8ae8a5a968bf0162
I20260925 14:54:21.375797 17380 catalog_manager.cc:4841] TS 6dd27596e14642608403942dc8260a88 (127.0.98.3:37291): tablet f257ca20ed81417a8ae8a5a968bf0162 (table test-table [id=1ac5d4e257db44f7802ffa5d7603d20f]) successfully deleted
I20260925 14:54:21.375845 17895 ts_tablet_manager.cc:1939] T f257ca20ed81417a8ae8a5a968bf0162 P 7596f711167b434087afcb75376fddc1: Deleting consensus metadata
I20260925 14:54:21.376526 17378 catalog_manager.cc:4841] TS 7596f711167b434087afcb75376fddc1 (127.0.98.2:43221): tablet f257ca20ed81417a8ae8a5a968bf0162 (table test-table [id=1ac5d4e257db44f7802ffa5d7603d20f]) successfully deleted
W20260925 14:54:21.376864 17378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37093 (error 108)
I20260925 14:54:21.377135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17590
I20260925 14:54:21.377449 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.377316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37093 (local address 127.0.0.1:48048)
0925 14:54:21.377366 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.377410 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 14:54:21.377699 17378 catalog_manager.cc:4584] TS 8d30c7f7db9c4cb3b27d7f47e4db6d8f (127.0.98.1:37093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f257ca20ed81417a8ae8a5a968bf0162: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37093: connect: Connection refused (error 111)
I20260925 14:54:21.382548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17724
I20260925 14:54:21.388638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17362
2026-09-25T14:54:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2095 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T14:54:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:21Z Disabled control of system clock
I20260925 14:54:21.415658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33645
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:21.496639 17900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:21.496853 17900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:21.496873 17900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:21.497943 17900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:21.497977 17900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:21.497989 17900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:21.498001 17900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:21.498014 17900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:21.499373 17900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:21.499565 17900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:21.499789 17900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:21.504576 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.504576 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.504671 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:21.504698 17900 server_base.cc:1034] running on GCE node
I20260925 14:54:21.505064 17900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:21.505367 17900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:21.506507 17900 hybrid_clock.cc:648] HybridClock initialized: now 1790348061506496 us; error 27 us; skew 500 ppm
I20260925 14:54:21.507771 17900 webserver.cc:460] Webserver started at http://127.0.98.62:38871/ using document root <none> and password file <none>
I20260925 14:54:21.508016 17900 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:21.508065 17900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:21.508237 17900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:21.509296 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "3cd840d7b96e452296222999f79734e2"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ae11a017377b2a247b6d02d0d941dce6"
server_key_iv: "0d6521de0c48a26c737090bf0316e212"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.509670 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cd840d7b96e452296222999f79734e2"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ae11a017377b2a247b6d02d0d941dce6"
server_key_iv: "0d6521de0c48a26c737090bf0316e212"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.511044 17900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:21.511788 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.511973 17900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:21.512080 17900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "3cd840d7b96e452296222999f79734e2"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ae11a017377b2a247b6d02d0d941dce6"
server_key_iv: "0d6521de0c48a26c737090bf0316e212"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.512153 17900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:21.530215 17900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:21.530633 17900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:21.530802 17900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:21.534825 17900 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33645
I20260925 14:54:21.534868 17966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33645 every 8 connection(s)
I20260925 14:54:21.535236 17900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:21.536182 17967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.538535 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: Bootstrap starting.
I20260925 14:54:21.539116 17967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.539420 17967 log.cc:824] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.539834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17900
I20260925 14:54:21.539958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:21.540151 392 external_mini_cluster.cc:1428] Setting key 843b8a3d1d51000e514728faf36bf6cc
I20260925 14:54:21.540218 17967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: No bootstrap required, opened a new log
I20260925 14:54:21.542857 17967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } }
I20260925 14:54:21.543066 17967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.543102 17967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd840d7b96e452296222999f79734e2, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.543222 17967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } }
I20260925 14:54:21.543299 17967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:21.543330 17967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:21.543372 17967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:21.544059 17967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } }
I20260925 14:54:21.544144 17967 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd840d7b96e452296222999f79734e2; no voters:
I20260925 14:54:21.544354 17967 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:21.544387 17972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:21.544497 17972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [term 1 LEADER]: Becoming Leader. State: Replica: 3cd840d7b96e452296222999f79734e2, State: Running, Role: LEADER
I20260925 14:54:21.544634 17967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:21.544634 17972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } }
I20260925 14:54:21.545075 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd840d7b96e452296222999f79734e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } } }
I20260925 14:54:21.545260 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:21.545735 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:21.545064 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd840d7b96e452296222999f79734e2. Latest consensus state: current_term: 1 leader_uuid: "3cd840d7b96e452296222999f79734e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd840d7b96e452296222999f79734e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33645 } } }
I20260925 14:54:21.545935 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:21.546312 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.540696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.98.62:33645)
0925 14:54:21.540887 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.540895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.540935 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:54:21.541499 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.541506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.541566 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.541685 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.541920 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.542837 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.543915 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.544453 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.544574 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.545304 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.545308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.545309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:21.545558 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.545602 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:21.545616 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.545737 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.546003 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.546086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260925 14:54:21.546331 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:21.547892 17976 catalog_manager.cc:1347] Generated new cluster ID: 9c420a54671842f58af978d780b379a7
I20260925 14:54:21.547950 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:21.560726 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:21.561456 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:21.564422 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: Generated new TSK 0
I20260925 14:54:21.564602 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:21.571153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--builtin_ntp_servers=127.0.98.20:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:21.654464 17991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:21.654692 17991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:21.654729 17991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:21.655983 17991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:21.656045 17991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:21.656086 17991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:21.657446 17991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:21.657713 17991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:21.658015 17991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:21.664074 17996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.664072 17997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.664183 17999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:21.664362 17991 server_base.cc:1034] running on GCE node
I20260925 14:54:21.664562 17991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:21.664831 17991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:21.666025 17991 hybrid_clock.cc:648] HybridClock initialized: now 1790348061666009 us; error 36 us; skew 500 ppm
I20260925 14:54:21.667424 17991 webserver.cc:460] Webserver started at http://127.0.98.1:42331/ using document root <none> and password file <none>
I20260925 14:54:21.667690 17991 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:21.667762 17991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:21.667912 17991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:21.668993 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce898662fa9d47ddbbce390a4224cc08"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ce0f8be73058e1356badd6b7ffca7f07"
server_key_iv: "1cfd11f88707b4b2011a687d3f5842f5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.669353 17991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce898662fa9d47ddbbce390a4224cc08"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ce0f8be73058e1356badd6b7ffca7f07"
server_key_iv: "1cfd11f88707b4b2011a687d3f5842f5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.670734 17991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:21.671522 18005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.671690 17991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:21.671788 17991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "ce898662fa9d47ddbbce390a4224cc08"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "ce0f8be73058e1356badd6b7ffca7f07"
server_key_iv: "1cfd11f88707b4b2011a687d3f5842f5"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.671871 17991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:21.685410 17991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:21.685784 17991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:21.685938 17991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:21.686175 17991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:21.686547 17991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:21.686600 17991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.686642 17991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:21.686669 17991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.692478 17991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42767
I20260925 14:54:21.692533 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42767 every 8 connection(s)
I20260925 14:54:21.692853 17991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:21.696022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 17991
I20260925 14:54:21.696131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:21.696354 392 external_mini_cluster.cc:1428] Setting key e425a1cd1a72cb1f4187fc9dd5e0552d
I20260925 14:54:21.697378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--builtin_ntp_servers=127.0.98.20:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:54:21.698529 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.693525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40117 (local address 127.0.98.62:33645)
0925 14:54:21.693600 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.693602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.693822 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:54:21.694292 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.694293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.694308 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.694436 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.695120 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.695603 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.697031 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.697258 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.697308 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.698240 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.698242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.698242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:21.698246 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.698260 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:21.698263 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.698305 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.698442 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.698473 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:54:21.698830 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.693525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33645 (local address 127.0.98.1:40117)
0925 14:54:21.693697 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.693732 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.694212 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.694532 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.694536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.694545 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.695046 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.695049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.695680 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.695681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.696881 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.696884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.697351 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.697352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.697536 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.698085 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:21.698086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.698126 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:21.698154 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:21.698325 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:21.698330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:21.698333 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.698429 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.698566 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260925 14:54:21.699332 18119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33645
I20260925 14:54:21.699422 18119 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:21.699623 18119 heartbeater.cc:511] Master 127.0.98.62:33645 requested a full tablet report, sending...
I20260925 14:54:21.700109 17931 ts_manager.cc:194] Registered new tserver with Master: ce898662fa9d47ddbbce390a4224cc08 (127.0.98.1:42767)
I20260925 14:54:21.700722 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40117
W20260925 14:54:21.779083 18122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:21.779368 18122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:21.779407 18122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:21.780572 18122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:21.780637 18122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:21.780682 18122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:21.782176 18122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:21.782462 18122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:21.782771 18122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:21.787830 18128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.787825 18130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.787865 18127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:21.788038 18122 server_base.cc:1034] running on GCE node
I20260925 14:54:21.788281 18122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:21.788491 18122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:21.789624 18122 hybrid_clock.cc:648] HybridClock initialized: now 1790348061789615 us; error 27 us; skew 500 ppm
I20260925 14:54:21.790915 18122 webserver.cc:460] Webserver started at http://127.0.98.2:37535/ using document root <none> and password file <none>
I20260925 14:54:21.791170 18122 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:21.791217 18122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:21.791385 18122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:21.792534 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "b03c84c0ffbc4c32a71804a52a518373"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "576ba5a9abe940a7dea89fef97e35463"
server_key_iv: "8317fc71f7d7840828628939730b47b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.792886 18122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b03c84c0ffbc4c32a71804a52a518373"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "576ba5a9abe940a7dea89fef97e35463"
server_key_iv: "8317fc71f7d7840828628939730b47b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.794134 18122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:21.794868 18136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.795028 18122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.795110 18122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "b03c84c0ffbc4c32a71804a52a518373"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "576ba5a9abe940a7dea89fef97e35463"
server_key_iv: "8317fc71f7d7840828628939730b47b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.795161 18122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:21.805197 18122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:21.805467 18122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:21.805581 18122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:21.805766 18122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:21.806109 18122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:21.806144 18122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.806200 18122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:21.806227 18122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.812024 18122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35605
I20260925 14:54:21.812129 18249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35605 every 8 connection(s)
I20260925 14:54:21.812462 18122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:21.817999 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.813294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55667 (local address 127.0.98.62:33645)
0925 14:54:21.813366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.813367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.813622 (+ 255us) server_negotiation.cc:407] Connection header received
0925 14:54:21.814132 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.814133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.814144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.814243 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.814950 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.815667 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.816620 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.816823 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.816880 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.817750 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.817751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.817751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:21.817755 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.817771 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:21.817775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.817830 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.817916 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.817938 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:21.818225 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.813319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33645 (local address 127.0.98.2:55667)
0925 14:54:21.813516 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.813551 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.814041 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.814306 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.814310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.814318 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.814857 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.814862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.815749 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.815750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.816542 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.816544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.816909 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.816910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.817014 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.817595 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:21.817595 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.817624 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:21.817646 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:21.817841 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:21.817845 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:21.817847 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.817911 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.817990 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260925 14:54:21.818761 18250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33645
I20260925 14:54:21.818864 18250 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:21.819016 18250 heartbeater.cc:511] Master 127.0.98.62:33645 requested a full tablet report, sending...
I20260925 14:54:21.819403 17931 ts_manager.cc:194] Registered new tserver with Master: b03c84c0ffbc4c32a71804a52a518373 (127.0.98.2:35605)
I20260925 14:54:21.819728 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55667
I20260925 14:54:21.822665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18122
I20260925 14:54:21.822747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:21.822911 392 external_mini_cluster.cc:1428] Setting key 7d418f8381c36a8df482b5c5bdc97e49
I20260925 14:54:21.823745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--builtin_ntp_servers=127.0.98.20:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:21.905221 18253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:21.905432 18253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:21.905453 18253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:21.906540 18253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:21.906572 18253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:21.906600 18253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:21.907855 18253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:21.908075 18253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:21.908380 18253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:21.913311 18259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.913311 18258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:21.913347 18261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:21.913549 18253 server_base.cc:1034] running on GCE node
I20260925 14:54:21.913774 18253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:21.914021 18253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:21.915186 18253 hybrid_clock.cc:648] HybridClock initialized: now 1790348061915173 us; error 26 us; skew 500 ppm
I20260925 14:54:21.916462 18253 webserver.cc:460] Webserver started at http://127.0.98.3:35763/ using document root <none> and password file <none>
I20260925 14:54:21.916702 18253 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:21.916747 18253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:21.916893 18253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:21.917994 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "5b9ddeaba18c410f9e97ec00b88354ea"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "c24be4ce5312cbfaf37e98594bc7bff8"
server_key_iv: "267cd7f736160fb5b4384928a5aa8153"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.918421 18253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5b9ddeaba18c410f9e97ec00b88354ea"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "c24be4ce5312cbfaf37e98594bc7bff8"
server_key_iv: "267cd7f736160fb5b4384928a5aa8153"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.919625 18253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:21.920289 18267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.920461 18253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:21.920557 18253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "5b9ddeaba18c410f9e97ec00b88354ea"
format_stamp: "Formatted at 2026-09-25 14:54:21 on dist-test-slave-gpc1"
server_key: "c24be4ce5312cbfaf37e98594bc7bff8"
server_key_iv: "267cd7f736160fb5b4384928a5aa8153"
server_key_version: "encryptionkey@0"
I20260925 14:54:21.920641 18253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:21.946074 18253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:21.946502 18253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:21.946674 18253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:21.946940 18253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:21.947327 18253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:21.947391 18253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.947443 18253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:21.947470 18253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:21.953553 18253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32921
I20260925 14:54:21.953605 18380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32921 every 8 connection(s)
I20260925 14:54:21.953995 18253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:21.958971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18253
I20260925 14:54:21.959100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:21.959312 392 external_mini_cluster.cc:1428] Setting key e861cee47938e1d0d954b27361ed95d2
I20260925 14:54:21.959446 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.954813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47227 (local address 127.0.98.62:33645)
0925 14:54:21.954879 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.954881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.955118 (+ 237us) server_negotiation.cc:407] Connection header received
0925 14:54:21.955645 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.955646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.955657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.955713 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.956409 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.956950 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.957937 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.958126 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.958188 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.959100 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.959101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.959102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:21.959107 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.959125 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:21.959130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.959184 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.959308 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.959336 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:21.959654 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.954802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33645 (local address 127.0.98.3:47227)
0925 14:54:21.954986 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.955022 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.955530 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.955778 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.955782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.955790 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:21.956308 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.956312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.957033 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.957033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.957835 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.957837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.958205 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.958206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.958313 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.958900 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:21.958901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:21.958948 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:21.958978 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:21.959219 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:21.959225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:21.959228 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.959310 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.959419 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260925 14:54:21.960116 18381 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33645
I20260925 14:54:21.960214 18381 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:21.960430 18381 heartbeater.cc:511] Master 127.0.98.62:33645 requested a full tablet report, sending...
I20260925 14:54:21.960769 17931 ts_manager.cc:194] Registered new tserver with Master: 5b9ddeaba18c410f9e97ec00b88354ea (127.0.98.3:32921)
I20260925 14:54:21.961190 17931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47227
I20260925 14:54:21.971329 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:21.975396 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.972780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.98.62:33645)
0925 14:54:21.972851 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.972853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.972870 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:54:21.972944 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.972945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.972954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.973009 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:21.973138 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.973618 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.974362 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.974542 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.974597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.975222 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:21.975223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:21.975223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:21.975227 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:21.975241 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:21.975244 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:21.975286 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.975320 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.975339 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:54:21.976949 17931 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:21.978538 17931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:21.985020 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.980665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32921 (local address 127.0.0.1:38544)
0925 14:54:21.980797 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.980837 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.981044 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.981337 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.981341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.981378 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.981451 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.981453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.982615 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.982616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.983783 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.983785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.984698 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.984777 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.984794 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.984872 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.984923 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260925 14:54:21.985145 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.980965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.0.1:54122)
0925 14:54:21.981148 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.981158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.981213 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.981615 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.981616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.981618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.981672 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.981673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.983051 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.983052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.984070 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.984072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.984853 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.984917 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.984921 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.984975 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.985023 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260925 14:54:21.985329 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.980760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.0.98.3:32921)
0925 14:54:21.980933 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.980938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.981005 (+ 67us) server_negotiation.cc:407] Connection header received
0925 14:54:21.981151 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.981154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.981175 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.981255 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.981543 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.982534 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.983873 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.984610 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984681 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.984771 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.985217 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.985257 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260925 14:54:21.985375 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.981160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35605 (local address 127.0.0.1:38498)
0925 14:54:21.981373 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:21.981382 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:21.981433 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:21.981859 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:21.981860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:21.981863 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:21.981899 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.981900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.983121 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.983122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.984152 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:21.984155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.985133 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.985134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:21.985194 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.985198 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:21.985254 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:54:21.985296 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":63,"threads_started":1}
I20260925 14:54:21.985469 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.981259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.0.98.1:42767)
0925 14:54:21.981449 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.981454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.981463 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:21.981502 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.981504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.981518 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.981750 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.981812 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.982946 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984179 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.984742 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984844 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.984935 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.985001 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.985043 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
I20260925 14:54:21.985641 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:21.981355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.0.98.2:35605)
0925 14:54:21.981622 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:21.981627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:21.981636 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:21.981680 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:21.981682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:21.981696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:21.981774 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:21.981982 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.983054 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.984244 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:21.984949 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:21.985019 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:21.985101 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:21.985511 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 14:54:21.985549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260925 14:54:21.986124 18315 tablet_service.cc:1467] Processing CreateTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 (DEFAULT_TABLE table=test-table [id=7b5039a923eb4bf5a07428ce058b612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.986123 18053 tablet_service.cc:1467] Processing CreateTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 (DEFAULT_TABLE table=test-table [id=7b5039a923eb4bf5a07428ce058b612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.986284 18184 tablet_service.cc:1467] Processing CreateTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 (DEFAULT_TABLE table=test-table [id=7b5039a923eb4bf5a07428ce058b612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:21.986514 18053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05b38cd94554540a4dde2540f5b4ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.986514 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05b38cd94554540a4dde2540f5b4ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.986691 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f05b38cd94554540a4dde2540f5b4ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:21.989331 18395 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Bootstrap starting.
I20260925 14:54:21.989817 18396 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: Bootstrap starting.
I20260925 14:54:21.990058 18395 tablet_bootstrap.cc:654] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.990432 18395 log.cc:824] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.990517 18396 tablet_bootstrap.cc:654] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.990808 18396 log.cc:824] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.991151 18397 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Bootstrap starting.
I20260925 14:54:21.991269 18395 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: No bootstrap required, opened a new log
I20260925 14:54:21.991341 18395 ts_tablet_manager.cc:1397] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:54:21.991703 18397 tablet_bootstrap.cc:654] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:21.991983 18397 log.cc:824] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:21.992517 18396 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: No bootstrap required, opened a new log
I20260925 14:54:21.992599 18396 ts_tablet_manager.cc:1397] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:54:21.992689 18397 tablet_bootstrap.cc:492] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: No bootstrap required, opened a new log
I20260925 14:54:21.992764 18397 ts_tablet_manager.cc:1397] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:21.993980 18395 raft_consensus.cc:348] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.994181 18395 raft_consensus.cc:374] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.994217 18395 raft_consensus.cc:729] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03c84c0ffbc4c32a71804a52a518373, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.994333 18395 consensus_queue.cc:260] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.994637 18395 ts_tablet_manager.cc:1428] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:54:21.994668 18250 heartbeater.cc:503] Master 127.0.98.62:33645 was elected leader, sending a full tablet report...
I20260925 14:54:21.994599 18396 raft_consensus.cc:348] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.995036 18396 raft_consensus.cc:374] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.995083 18396 raft_consensus.cc:729] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce898662fa9d47ddbbce390a4224cc08, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.995191 18396 consensus_queue.cc:260] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.995290 18397 raft_consensus.cc:348] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.995452 18396 ts_tablet_manager.cc:1428] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:21.995456 18397 raft_consensus.cc:374] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:21.995496 18397 raft_consensus.cc:729] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9ddeaba18c410f9e97ec00b88354ea, State: Initialized, Role: FOLLOWER
I20260925 14:54:21.995515 18119 heartbeater.cc:503] Master 127.0.98.62:33645 was elected leader, sending a full tablet report...
I20260925 14:54:21.995611 18397 consensus_queue.cc:260] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:21.995863 18397 ts_tablet_manager.cc:1428] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:21.995934 18381 heartbeater.cc:503] Master 127.0.98.62:33645 was elected leader, sending a full tablet report...
I20260925 14:54:22.011688 18401 raft_consensus.cc:482] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:22.011858 18401 raft_consensus.cc:504] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:22.012217 18401 leader_election.cc:288] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b9ddeaba18c410f9e97ec00b88354ea (127.0.98.3:32921), ce898662fa9d47ddbbce390a4224cc08 (127.0.98.1:42767)
I20260925 14:54:22.015236 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.012610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55591 (local address 127.0.98.1:42767)
0925 14:54:22.012654 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.012656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.012757 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:54:22.012806 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.012807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.012819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.012893 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:22.013060 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.013546 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014488 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.014872 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014924 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.014933 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:22.015113 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 14:54:22.015160 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:22.015249 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.012420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32921 (local address 127.0.98.2:42945)
0925 14:54:22.012486 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.012502 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:22.012561 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:22.012704 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:22.012705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:22.012707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:22.012795 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.012797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.013515 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.013516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.014315 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.014318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.014998 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.015066 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.015069 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:22.015143 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:54:22.015181 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:54:22.015271 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.012510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42945 (local address 127.0.98.3:32921)
0925 14:54:22.012558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.012560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.012570 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:22.012629 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.012630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.012640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.012693 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:22.012891 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.013437 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014495 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.014896 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014978 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.014987 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:22.015163 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 14:54:22.015209 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:54:22.015252 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.012542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.98.2:55591)
0925 14:54:22.012691 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.012701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:22.012752 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:22.012904 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:22.012905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:22.012906 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:22.012987 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.012989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.013623 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.013624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.014392 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:22.014395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.014938 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.014940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:22.015041 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.015045 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:22.015088 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:54:22.015151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260925 14:54:22.015841 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05b38cd94554540a4dde2540f5b4ae5" candidate_uuid: "b03c84c0ffbc4c32a71804a52a518373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce898662fa9d47ddbbce390a4224cc08" is_pre_election: true
I20260925 14:54:22.015874 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05b38cd94554540a4dde2540f5b4ae5" candidate_uuid: "b03c84c0ffbc4c32a71804a52a518373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" is_pre_election: true
I20260925 14:54:22.016014 18073 raft_consensus.cc:2454] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03c84c0ffbc4c32a71804a52a518373 in term 0.
I20260925 14:54:22.016029 18335 raft_consensus.cc:2454] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b03c84c0ffbc4c32a71804a52a518373 in term 0.
I20260925 14:54:22.016254 18139 leader_election.cc:302] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9ddeaba18c410f9e97ec00b88354ea, b03c84c0ffbc4c32a71804a52a518373; no voters:
I20260925 14:54:22.016428 18401 raft_consensus.cc:2781] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:22.016484 18401 raft_consensus.cc:482] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:22.016506 18401 raft_consensus.cc:3037] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:22.017212 18401 raft_consensus.cc:504] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:22.017354 18401 leader_election.cc:288] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [CANDIDATE]: Term 1 election: Requested vote from peers 5b9ddeaba18c410f9e97ec00b88354ea (127.0.98.3:32921), ce898662fa9d47ddbbce390a4224cc08 (127.0.98.1:42767)
I20260925 14:54:22.017508 18335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05b38cd94554540a4dde2540f5b4ae5" candidate_uuid: "b03c84c0ffbc4c32a71804a52a518373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9ddeaba18c410f9e97ec00b88354ea"
I20260925 14:54:22.017593 18073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f05b38cd94554540a4dde2540f5b4ae5" candidate_uuid: "b03c84c0ffbc4c32a71804a52a518373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce898662fa9d47ddbbce390a4224cc08"
I20260925 14:54:22.017611 18335 raft_consensus.cc:3037] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:22.017668 18073 raft_consensus.cc:3037] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:22.018565 18335 raft_consensus.cc:2454] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03c84c0ffbc4c32a71804a52a518373 in term 1.
I20260925 14:54:22.018560 18073 raft_consensus.cc:2454] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b03c84c0ffbc4c32a71804a52a518373 in term 1.
I20260925 14:54:22.018720 18139 leader_election.cc:302] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b03c84c0ffbc4c32a71804a52a518373, ce898662fa9d47ddbbce390a4224cc08; no voters:
I20260925 14:54:22.018824 18401 raft_consensus.cc:2781] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:22.018981 18401 raft_consensus.cc:686] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 1 LEADER]: Becoming Leader. State: Replica: b03c84c0ffbc4c32a71804a52a518373, State: Running, Role: LEADER
I20260925 14:54:22.019089 18401 consensus_queue.cc:237] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } }
I20260925 14:54:22.019965 17930 catalog_manager.cc:5495] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 reported cstate change: term changed from 0 to 1, leader changed from <none> to b03c84c0ffbc4c32a71804a52a518373 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b03c84c0ffbc4c32a71804a52a518373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b03c84c0ffbc4c32a71804a52a518373" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35605 } health_report { overall_health: HEALTHY } } }
I20260925 14:54:22.030714 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.028268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.98.2:35605)
0925 14:54:22.028329 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:22.028331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:22.028350 (+ 19us) server_negotiation.cc:407] Connection header received
0925 14:54:22.028419 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:22.028420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:22.028447 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:22.028533 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:22.028655 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.029125 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.029880 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:22.030078 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:22.030156 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:22.030307 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:22.030466 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:22.030557 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:22.030582 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:54:22.030602 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:22.032609 18073 raft_consensus.cc:1261] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 1 FOLLOWER]: Refusing update from remote peer b03c84c0ffbc4c32a71804a52a518373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:22.032620 18335 raft_consensus.cc:1261] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 1 FOLLOWER]: Refusing update from remote peer b03c84c0ffbc4c32a71804a52a518373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:22.032986 18401 consensus_queue.cc:1035] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b9ddeaba18c410f9e97ec00b88354ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:22.033131 18405 consensus_queue.cc:1035] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce898662fa9d47ddbbce390a4224cc08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:22.035669 18408 mvcc.cc:205] Tried to move back new op lower bound from 7333265662082523136 to 7333265662031171584. Current Snapshot: MvccSnapshot[applied={T|T < 7333265662082523136}]
I20260925 14:54:22.036170 18410 mvcc.cc:205] Tried to move back new op lower bound from 7333265662082523136 to 7333265662031171584. Current Snapshot: MvccSnapshot[applied={T|T < 7333265662082523136}]
I20260925 14:54:22.037819 17931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:22.037919 17931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:22.038632 17931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cd840d7b96e452296222999f79734e2: Sending DeleteTablet for 3 replicas of tablet f05b38cd94554540a4dde2540f5b4ae5
I20260925 14:54:22.038938 18315 tablet_service.cc:1514] Processing DeleteTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:22 UTC) from {username='slave'} at 127.0.0.1:38544
I20260925 14:54:22.038970 18053 tablet_service.cc:1514] Processing DeleteTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:22 UTC) from {username='slave'} at 127.0.0.1:54122
I20260925 14:54:22.039188 18419 tablet_replica.cc:330] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08: stopping tablet replica
I20260925 14:54:22.039178 18184 tablet_service.cc:1514] Processing DeleteTablet for tablet f05b38cd94554540a4dde2540f5b4ae5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:22 UTC) from {username='slave'} at 127.0.0.1:38498
I20260925 14:54:22.039283 18419 raft_consensus.cc:2229] T f05b38cd94554540a4dde2540f5b4ae5 P ce898662fa9d47ddbbce390a4224cc08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:22.039376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17991
I20260925 14:54:22.040481 18420 tablet_replica.cc:330] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: stopping tablet replica
I20260925 14:54:22.040596 18420 raft_consensus.cc:2229] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:22.040709 18420 raft_consensus.cc:2258] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:22.040871 18418 tablet_replica.cc:330] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: stopping tablet replica
I20260925 14:54:22.041079 18418 raft_consensus.cc:2229] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:22.041182 18418 raft_consensus.cc:2258] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:22.041280 18420 ts_tablet_manager.cc:1905] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:22.041604 18418 ts_tablet_manager.cc:1905] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:22.042871 18420 ts_tablet_manager.cc:1918] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:22.042948 18420 log.cc:1196] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/f05b38cd94554540a4dde2540f5b4ae5
I20260925 14:54:22.043169 18418 ts_tablet_manager.cc:1918] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:22.043223 18420 ts_tablet_manager.cc:1939] T f05b38cd94554540a4dde2540f5b4ae5 P b03c84c0ffbc4c32a71804a52a518373: Deleting consensus metadata
I20260925 14:54:22.043237 18418 log.cc:1196] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/f05b38cd94554540a4dde2540f5b4ae5
I20260925 14:54:22.043495 18418 ts_tablet_manager.cc:1939] T f05b38cd94554540a4dde2540f5b4ae5 P 5b9ddeaba18c410f9e97ec00b88354ea: Deleting consensus metadata
I20260925 14:54:22.043614 17917 catalog_manager.cc:4841] TS b03c84c0ffbc4c32a71804a52a518373 (127.0.98.2:35605): tablet f05b38cd94554540a4dde2540f5b4ae5 (table test-table [id=7b5039a923eb4bf5a07428ce058b612a]) successfully deleted
I20260925 14:54:22.043872 17917 catalog_manager.cc:4841] TS 5b9ddeaba18c410f9e97ec00b88354ea (127.0.98.3:32921): tablet f05b38cd94554540a4dde2540f5b4ae5 (table test-table [id=7b5039a923eb4bf5a07428ce058b612a]) successfully deleted
W20260925 14:54:22.044785 17917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42767 (error 108)
I20260925 14:54:22.045281 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:22.045117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42767 (local address 127.0.0.1:54136)
0925 14:54:22.045180 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:22.045241 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:54:22.045305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18122
W20260925 14:54:22.046739 17917 catalog_manager.cc:4584] TS ce898662fa9d47ddbbce390a4224cc08 (127.0.98.1:42767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f05b38cd94554540a4dde2540f5b4ae5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42767: connect: Connection refused (error 111)
I20260925 14:54:22.050529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18253
I20260925 14:54:22.055346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 17900
2026-09-25T14:54:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:22 dist-test-slave-gpc1 krb5kdc[18424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:22 dist-test-slave-gpc1 krb5kdc[18424](info): set up 2 sockets
Sep 25 14:54:22 dist-test-slave-gpc1 krb5kdc[18424](info): commencing operation
krb5kdc: starting...
W20260925 14:54:23.126176 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:23.240707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:23.322613 18437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:23.322850 18437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:23.322898 18437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:23.324071 18437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:23.324137 18437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:23.324165 18437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:23.324195 18437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:23.324221 18437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:23.325780 18437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:23.326076 18437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:23.326296 18437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:23.331087 18443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.331264 18437 server_base.cc:1034] running on GCE node
W20260925 14:54:23.331090 18442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:23.331066 18445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.331606 18437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:23.331856 18437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:23.333002 18437 hybrid_clock.cc:648] HybridClock initialized: now 1790348063332991 us; error 26 us; skew 500 ppm
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:23.335419 18437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:23.335786 18437 webserver.cc:460] Webserver started at http://127.0.98.62:45057/ using document root <none> and password file <none>
I20260925 14:54:23.335980 18437 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:23.336020 18437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:23.336109 18437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:23.337095 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "0ba232862faa473aa4f21c07e3ff9af3"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "0943693d87f1fab2159b1dbd279fe52d"
server_key_iv: "934de230c32ae1eac60af0fcd662f61d"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.337390 18437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ba232862faa473aa4f21c07e3ff9af3"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "0943693d87f1fab2159b1dbd279fe52d"
server_key_iv: "934de230c32ae1eac60af0fcd662f61d"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.338635 18437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:23.339598 18452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.339828 18437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:23.339920 18437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0ba232862faa473aa4f21c07e3ff9af3"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "0943693d87f1fab2159b1dbd279fe52d"
server_key_iv: "934de230c32ae1eac60af0fcd662f61d"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.339978 18437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:23.360220 18437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:23.362040 18437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:23.362149 18437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:23.365967 18437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42329
I20260925 14:54:23.366021 18504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42329 every 8 connection(s)
I20260925 14:54:23.366380 18437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:23.367311 18505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:23.369561 18505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: Bootstrap starting.
I20260925 14:54:23.370148 18505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:23.370494 18505 log.cc:824] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:23.371176 18505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: No bootstrap required, opened a new log
I20260925 14:54:23.372730 18505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260925 14:54:23.372910 18505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:23.372959 18505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba232862faa473aa4f21c07e3ff9af3, State: Initialized, Role: FOLLOWER
I20260925 14:54:23.373154 18505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260925 14:54:23.373258 18505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:23.373297 18505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:23.373350 18505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:23.373870 18505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260925 14:54:23.373986 18505 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ba232862faa473aa4f21c07e3ff9af3; no voters:
I20260925 14:54:23.374168 18505 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:23.374221 18508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:23.374457 18505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:23.374491 18508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba232862faa473aa4f21c07e3ff9af3, State: Running, Role: LEADER
I20260925 14:54:23.374572 18508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } }
I20260925 14:54:23.375056 18510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba232862faa473aa4f21c07e3ff9af3. Latest consensus state: current_term: 1 leader_uuid: "0ba232862faa473aa4f21c07e3ff9af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } } }
I20260925 14:54:23.375180 18510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:23.375257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18437
I20260925 14:54:23.375391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:23.375357 18509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ba232862faa473aa4f21c07e3ff9af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ba232862faa473aa4f21c07e3ff9af3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42329 } } }
I20260925 14:54:23.375582 18509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:23.375602 392 external_mini_cluster.cc:1428] Setting key 23694317addbd0983fb137970db5cf07
I20260925 14:54:23.375766 18519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:23.376355 18519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:23.377918 18519 catalog_manager.cc:1347] Generated new cluster ID: 02d7aafe825e46cf955ae162af59c68b
I20260925 14:54:23.377972 18519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:23.384855 18519 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:23.385511 18519 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:23.386910 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.376195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.0.98.62:42329)
0925 14:54:23.376362 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.376371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.376396 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:54:23.376526 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.376533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.376589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.376731 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.377492 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.378891 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.379999 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.380585 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.380676 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.383449 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:23.383455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:23.383455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:23.383471 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:23.385443 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.385790 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.385793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.385793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.385883 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.386094 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.386095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.386096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.386300 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:23.386500 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.386595 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.386746 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260925 14:54:23.389266 18519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: Generated new TSK 0
I20260925 14:54:23.389413 18519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:54:23.419389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:23.509876 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:23.510088 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:23.510113 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:23.511404 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:23.511451 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:23.511477 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:23.512981 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:23.513221 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:23.513470 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:23.519564 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:23.519518 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.519667 18531 server_base.cc:1034] running on GCE node
W20260925 14:54:23.519526 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.520062 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:23.520346 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:23.521514 18531 hybrid_clock.cc:648] HybridClock initialized: now 1790348063521496 us; error 33 us; skew 500 ppm
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:23.524531 18531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:54:23.525035 18531 webserver.cc:460] Webserver started at http://127.0.98.1:39231/ using document root <none> and password file <none>
I20260925 14:54:23.525273 18531 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:23.525341 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:23.525483 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:23.526633 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "622a0f68b40c484bbdd3811b2f44441a"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "66a8e52f9438bf9c4d990f572a9a9dc3"
server_key_iv: "c7a9303a1e700452138d7f1eb82eb912"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.527019 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "622a0f68b40c484bbdd3811b2f44441a"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "66a8e52f9438bf9c4d990f572a9a9dc3"
server_key_iv: "c7a9303a1e700452138d7f1eb82eb912"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.528441 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:23.529162 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.529325 18531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:23.529433 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "622a0f68b40c484bbdd3811b2f44441a"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "66a8e52f9438bf9c4d990f572a9a9dc3"
server_key_iv: "c7a9303a1e700452138d7f1eb82eb912"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.529508 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:23.539665 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:23.541505 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:23.541664 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:23.541913 18531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:23.542222 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:23.542277 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.542321 18531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:23.542367 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.548223 18531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33031
I20260925 14:54:23.548265 18659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33031 every 8 connection(s)
I20260925 14:54:23.548573 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:23.554550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18531
I20260925 14:54:23.554680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:23.554881 392 external_mini_cluster.cc:1428] Setting key 4c82cf05be1295b667b3257d00b0b7e9
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:23.558564 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.549248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58459 (local address 127.0.98.62:42329)
0925 14:54:23.549315 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.549317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.549528 (+ 211us) server_negotiation.cc:407] Connection header received
0925 14:54:23.550518 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.550519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.550533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.550584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.551197 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.551710 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.552716 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.552950 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.553034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.555487 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:23.555489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:23.555489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:23.555497 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:23.557051 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.557454 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.557456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.557456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.557525 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.557754 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.557754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.557755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.557910 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:23.557997 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.558370 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.558474 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:23.558768 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.549228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42329 (local address 127.0.98.1:58459)
0925 14:54:23.549413 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.549448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.550408 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.550659 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.550662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.550913 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.551100 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.551104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.551806 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.551807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.552624 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.552626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.553045 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.553046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.553164 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.553693 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.553701 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.555366 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:23.557138 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.557144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.557146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.557381 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.557592 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.557592 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.557593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.557692 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.558031 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:23.558042 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:23.558206 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.558311 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.558521 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260925 14:54:23.559280 18660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42329
I20260925 14:54:23.559409 18660 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:23.559623 18660 heartbeater.cc:511] Master 127.0.98.62:42329 requested a full tablet report, sending...
I20260925 14:54:23.560163 18469 ts_manager.cc:194] Registered new tserver with Master: 622a0f68b40c484bbdd3811b2f44441a (127.0.98.1:33031)
I20260925 14:54:23.561172 18469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58459
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:54:23.580346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:23.662882 18665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:23.663096 18665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:23.663121 18665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:23.664381 18665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:23.664427 18665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:23.664454 18665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:23.665961 18665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:23.666213 18665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:23.666500 18665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:23.671556 18671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:23.671569 18670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.671588 18665 server_base.cc:1034] running on GCE node
W20260925 14:54:23.671569 18673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.672013 18665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:23.672293 18665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:23.673494 18665 hybrid_clock.cc:648] HybridClock initialized: now 1790348063673462 us; error 42 us; skew 500 ppm
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:23.676268 18665 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:54:23.676767 18665 webserver.cc:460] Webserver started at http://127.0.98.2:39281/ using document root <none> and password file <none>
I20260925 14:54:23.677004 18665 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:23.677075 18665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:23.677235 18665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:23.678256 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "842660bc6ac9498a979266d8d47c6345"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "1415164e756545ae53172761469e4dca"
server_key_iv: "3292fbb9062c139111cfc61e7528ba24"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.678689 18665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "842660bc6ac9498a979266d8d47c6345"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "1415164e756545ae53172761469e4dca"
server_key_iv: "3292fbb9062c139111cfc61e7528ba24"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.679962 18665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:23.680660 18680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.680823 18665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:23.680922 18665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "842660bc6ac9498a979266d8d47c6345"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "1415164e756545ae53172761469e4dca"
server_key_iv: "3292fbb9062c139111cfc61e7528ba24"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.681006 18665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:23.709713 18665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:23.711701 18665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:23.711880 18665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:23.712127 18665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:23.712467 18665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:23.712522 18665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.712558 18665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:23.712584 18665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.718487 18665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33577
I20260925 14:54:23.718562 18793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33577 every 8 connection(s)
I20260925 14:54:23.718896 18665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:23.725292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18665
I20260925 14:54:23.725415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:23.725674 392 external_mini_cluster.cc:1428] Setting key 3e3f3c645f4f6f84793d0d4b6cb467e0
I20260925 14:54:23.728732 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.719604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51477 (local address 127.0.98.62:42329)
0925 14:54:23.719650 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.719652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.719882 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:54:23.720879 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.720880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.720892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.720937 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.721584 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.722088 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.723132 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.723321 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.723402 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.725595 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:23.725596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:23.725596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:23.725603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:23.727081 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.727577 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.727597 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.727598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.727666 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.728004 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.728005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.728006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.728148 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:23.728213 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.728589 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.728667 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:23.728982 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.719570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42329 (local address 127.0.98.2:51477)
0925 14:54:23.719748 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.719805 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.720763 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.721013 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.721018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.721289 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.721478 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.721483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.722183 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.722184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.723027 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.723029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.723411 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.723412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.723541 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.724013 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.724022 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.725505 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:23.727164 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.727171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.727173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.727500 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.727757 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.727758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.727759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.727909 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.728225 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:23.728232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:23.728425 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.728551 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.728740 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260925 14:54:23.729461 18794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42329
I20260925 14:54:23.729555 18794 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:23.729753 18794 heartbeater.cc:511] Master 127.0.98.62:42329 requested a full tablet report, sending...
I20260925 14:54:23.730150 18469 ts_manager.cc:194] Registered new tserver with Master: 842660bc6ac9498a979266d8d47c6345 (127.0.98.2:33577)
I20260925 14:54:23.730594 18469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51477
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:54:23.750497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:23.831499 18799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:23.831701 18799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:23.831725 18799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:23.832896 18799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:23.832937 18799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:23.832970 18799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:23.834470 18799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:23.834710 18799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:23.834913 18799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:23.839768 18807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:23.839775 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:23.839803 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:23.840189 18799 server_base.cc:1034] running on GCE node
I20260925 14:54:23.840413 18799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:23.840665 18799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:23.841807 18799 hybrid_clock.cc:648] HybridClock initialized: now 1790348063841790 us; error 37 us; skew 500 ppm
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:23.844485 18799 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:54:23.844982 18799 webserver.cc:460] Webserver started at http://127.0.98.3:43495/ using document root <none> and password file <none>
I20260925 14:54:23.845177 18799 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:23.845217 18799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:23.845306 18799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:23.846297 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "587a68bb03a8451ea81365b5b05c14fd"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "3b4e785fa88154a76b5c71f1cecd71d8"
server_key_iv: "9428ce5796eb2622219168b78058c873"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.846683 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "587a68bb03a8451ea81365b5b05c14fd"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "3b4e785fa88154a76b5c71f1cecd71d8"
server_key_iv: "9428ce5796eb2622219168b78058c873"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.847963 18799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:23.848603 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.848769 18799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:23.848872 18799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "587a68bb03a8451ea81365b5b05c14fd"
format_stamp: "Formatted at 2026-09-25 14:54:23 on dist-test-slave-gpc1"
server_key: "3b4e785fa88154a76b5c71f1cecd71d8"
server_key_iv: "9428ce5796eb2622219168b78058c873"
server_key_version: "encryptionkey@0"
I20260925 14:54:23.848948 18799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:23.878535 18799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:23.880610 18799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:23.880781 18799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:23.881031 18799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:23.881349 18799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:23.881402 18799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.881446 18799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:23.881469 18799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:23.887730 18799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37635
I20260925 14:54:23.887773 18927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37635 every 8 connection(s)
I20260925 14:54:23.888108 18799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:54:23 dist-test-slave-gpc1 krb5kdc[18424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:23.895670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18799
I20260925 14:54:23.895792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:23.896020 392 external_mini_cluster.cc:1428] Setting key 1164527582ab7e8d41765bdbe4e75bf2
I20260925 14:54:23.897751 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.888838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54407 (local address 127.0.98.62:42329)
0925 14:54:23.888886 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.888892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.889105 (+ 213us) server_negotiation.cc:407] Connection header received
0925 14:54:23.890127 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.890128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.890138 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.890204 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.890876 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.891374 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.892400 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.892591 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.892649 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.894749 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:23.894750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:23.894750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:23.894757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:23.896285 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.896750 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.896751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.896752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.896819 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.897097 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.897098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.897099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.897219 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:23.897294 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.897574 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.897663 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:23.897925 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.888788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42329 (local address 127.0.98.3:54407)
0925 14:54:23.888946 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.888999 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.890032 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.890263 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.890267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.890598 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:23.890795 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.890799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.891458 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.891459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.892316 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.892318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.892668 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.892669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.892778 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.893290 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:23.893298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:23.894639 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:23.896361 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.896368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.896371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.896659 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.896894 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:23.896894 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:23.896895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:23.897021 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:23.897285 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:23.897292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:23.897434 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.897539 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.897693 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 14:54:23.898321 18928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42329
I20260925 14:54:23.898469 18928 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:23.898643 18928 heartbeater.cc:511] Master 127.0.98.62:42329 requested a full tablet report, sending...
I20260925 14:54:23.899030 18469 ts_manager.cc:194] Registered new tserver with Master: 587a68bb03a8451ea81365b5b05c14fd (127.0.98.3:37635)
I20260925 14:54:23.899432 18469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54407
I20260925 14:54:23.908037 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:23.915666 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.909375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.0.98.62:42329)
0925 14:54:23.909444 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.909447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.909525 (+ 78us) server_negotiation.cc:407] Connection header received
0925 14:54:23.909575 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.909575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.909584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.909630 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:23.910140 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.910676 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.911522 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.911695 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.911743 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.913044 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:23.913045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:23.913045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:23.913051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:23.914496 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.914877 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.914878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.914878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.914927 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:23.915109 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:23.915109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:23.915110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:23.915226 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:23.915302 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.915492 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.915541 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:23.917095 18469 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:23.918707 18469 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:23.925486 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33031 (local address 127.0.0.1:46550)
0925 14:54:23.921799 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.921808 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.921859 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.922109 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.922110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.922112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:23.922154 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.922155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.923068 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.923070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.924295 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.924298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.925197 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.925198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.925278 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.925295 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.925361 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.925412 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":93,"threads_started":1}
I20260925 14:54:23.925588 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.0.98.3:37635)
0925 14:54:23.921509 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.921512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.921607 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:54:23.921679 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.921681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.921695 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.922147 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:23.922231 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.923307 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.924540 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.925207 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.925303 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.925359 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.925461 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.925505 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260925 14:54:23.925943 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.0.98.1:33031)
0925 14:54:23.921461 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.921466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.921853 (+ 387us) server_negotiation.cc:407] Connection header received
0925 14:54:23.921928 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.921931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.921946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.922017 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:23.922247 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.922971 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.924378 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.925109 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.925186 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.925251 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.925822 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.925868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260925 14:54:23.925590 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37635 (local address 127.0.0.1:43588)
0925 14:54:23.921497 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.921535 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.921605 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.921987 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.921990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.921995 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:23.922062 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.922064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.923413 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.923414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.924459 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.924460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.925312 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.925313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.925380 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.925384 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.925440 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.925485 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"spinlock_wait_cycles":896,"thread_start_us":116,"threads_started":1}
I20260925 14:54:23.926515 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33577 (local address 127.0.0.1:37590)
0925 14:54:23.921678 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:23.921686 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:23.921718 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:23.921987 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:23.921990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:23.921992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:23.922062 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.922065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.924537 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.924537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.925225 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:23.925227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.926274 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.926275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:23.926361 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.926364 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:23.926415 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:54:23.926452 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":6,"thread_start_us":51,"threads_started":1}
I20260925 14:54:23.926795 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:23.921216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.0.98.2:33577)
0925 14:54:23.921397 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:23.921402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:23.921741 (+ 339us) server_negotiation.cc:407] Connection header received
0925 14:54:23.921785 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:23.921787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:23.921802 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:23.921876 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:23.922149 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.923307 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.925323 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:23.926105 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:23.926166 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:23.926227 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:23.926677 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 14:54:23.926722 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260925 14:54:23.926834 18594 tablet_service.cc:1467] Processing CreateTablet for tablet a72ce782730a4a909489b82297952e8a (DEFAULT_TABLE table=test-table [id=16a8fd7f436c4d0da5a712b55bf22300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:23.927225 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72ce782730a4a909489b82297952e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:23.927407 18728 tablet_service.cc:1467] Processing CreateTablet for tablet a72ce782730a4a909489b82297952e8a (DEFAULT_TABLE table=test-table [id=16a8fd7f436c4d0da5a712b55bf22300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:23.927699 18728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72ce782730a4a909489b82297952e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:23.930011 18945 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Bootstrap starting.
I20260925 14:54:23.930410 18946 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Bootstrap starting.
I20260925 14:54:23.930476 18862 tablet_service.cc:1467] Processing CreateTablet for tablet a72ce782730a4a909489b82297952e8a (DEFAULT_TABLE table=test-table [id=16a8fd7f436c4d0da5a712b55bf22300]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:23.930852 18862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72ce782730a4a909489b82297952e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:23.930935 18945 tablet_bootstrap.cc:654] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:23.931088 18946 tablet_bootstrap.cc:654] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:23.931488 18945 log.cc:824] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:23.931602 18946 log.cc:824] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:23.932325 18946 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: No bootstrap required, opened a new log
I20260925 14:54:23.932405 18946 ts_tablet_manager.cc:1397] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:23.933984 18949 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Bootstrap starting.
I20260925 14:54:23.934032 18946 raft_consensus.cc:348] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.934327 18946 raft_consensus.cc:374] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:23.934403 18946 raft_consensus.cc:729] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842660bc6ac9498a979266d8d47c6345, State: Initialized, Role: FOLLOWER
I20260925 14:54:23.934542 18946 consensus_queue.cc:260] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.934625 18945 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: No bootstrap required, opened a new log
I20260925 14:54:23.934685 18945 ts_tablet_manager.cc:1397] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260925 14:54:23.934705 18949 tablet_bootstrap.cc:654] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:23.934832 18946 ts_tablet_manager.cc:1428] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 14:54:23.934959 18794 heartbeater.cc:503] Master 127.0.98.62:42329 was elected leader, sending a full tablet report...
I20260925 14:54:23.935302 18949 log.cc:824] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:23.936012 18949 tablet_bootstrap.cc:492] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: No bootstrap required, opened a new log
I20260925 14:54:23.936090 18949 ts_tablet_manager.cc:1397] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:23.936637 18945 raft_consensus.cc:348] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.936770 18945 raft_consensus.cc:374] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:23.936792 18945 raft_consensus.cc:729] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a0f68b40c484bbdd3811b2f44441a, State: Initialized, Role: FOLLOWER
I20260925 14:54:23.936893 18945 consensus_queue.cc:260] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.937181 18945 ts_tablet_manager.cc:1428] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:54:23.937222 18660 heartbeater.cc:503] Master 127.0.98.62:42329 was elected leader, sending a full tablet report...
I20260925 14:54:23.938236 18949 raft_consensus.cc:348] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.938434 18949 raft_consensus.cc:374] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:23.938488 18949 raft_consensus.cc:729] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587a68bb03a8451ea81365b5b05c14fd, State: Initialized, Role: FOLLOWER
I20260925 14:54:23.938612 18949 consensus_queue.cc:260] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:23.938918 18949 ts_tablet_manager.cc:1428] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:23.938934 18928 heartbeater.cc:503] Master 127.0.98.62:42329 was elected leader, sending a full tablet report...
W20260925 14:54:23.969437 18795 tablet.cc:2369] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:24.049316 18661 tablet.cc:2369] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:54:24.138651 18929 tablet.cc:2369] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:24.321743 18950 raft_consensus.cc:482] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:24.321872 18950 raft_consensus.cc:504] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:24.322305 18950 leader_election.cc:288] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 587a68bb03a8451ea81365b5b05c14fd (127.0.98.3:37635), 622a0f68b40c484bbdd3811b2f44441a (127.0.98.1:33031)
I20260925 14:54:24.325366 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.322668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36755 (local address 127.0.98.1:33031)
0925 14:54:24.322709 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.322711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.322829 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:54:24.322898 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.322899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.322911 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.322991 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:24.323162 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.323643 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.324644 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.325012 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.325066 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.325097 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.325250 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.325294 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:54:24.325385 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.322643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36299 (local address 127.0.98.3:37635)
0925 14:54:24.322711 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.322713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.322822 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:54:24.322897 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.322898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.322908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.322998 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:24.323202 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.323642 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.324645 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.325021 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.325109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.325118 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.325285 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.325325 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:54:24.325376 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.322588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33031 (local address 127.0.98.2:36755)
0925 14:54:24.322735 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.322746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.322838 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.322997 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.322998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.323000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:24.323096 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.323098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.323730 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.323731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.324549 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.324551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.325097 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.325098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.325171 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.325175 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:24.325226 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:54:24.325283 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":23,"thread_start_us":74,"threads_started":1}
I20260925 14:54:24.325395 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.322535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37635 (local address 127.0.98.2:36299)
0925 14:54:24.322735 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.322746 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.322833 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.323005 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.323005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.323007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:24.323111 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.323113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.323729 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.323730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.324551 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.324553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.325124 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.325125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.325195 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.325198 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:24.325270 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:54:24.325303 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":23,"thread_start_us":102,"threads_started":1}
I20260925 14:54:24.325742 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72ce782730a4a909489b82297952e8a" candidate_uuid: "842660bc6ac9498a979266d8d47c6345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a0f68b40c484bbdd3811b2f44441a" is_pre_election: true
I20260925 14:54:24.325927 18614 raft_consensus.cc:2454] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842660bc6ac9498a979266d8d47c6345 in term 0.
I20260925 14:54:24.326000 18882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72ce782730a4a909489b82297952e8a" candidate_uuid: "842660bc6ac9498a979266d8d47c6345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587a68bb03a8451ea81365b5b05c14fd" is_pre_election: true
I20260925 14:54:24.326143 18882 raft_consensus.cc:2454] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 842660bc6ac9498a979266d8d47c6345 in term 0.
I20260925 14:54:24.326161 18683 leader_election.cc:302] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622a0f68b40c484bbdd3811b2f44441a, 842660bc6ac9498a979266d8d47c6345; no voters:
I20260925 14:54:24.326295 18950 raft_consensus.cc:2781] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:24.326377 18950 raft_consensus.cc:482] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:24.326413 18950 raft_consensus.cc:3037] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:24.327121 18950 raft_consensus.cc:504] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:24.327260 18950 leader_election.cc:288] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [CANDIDATE]: Term 1 election: Requested vote from peers 587a68bb03a8451ea81365b5b05c14fd (127.0.98.3:37635), 622a0f68b40c484bbdd3811b2f44441a (127.0.98.1:33031)
I20260925 14:54:24.327428 18882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72ce782730a4a909489b82297952e8a" candidate_uuid: "842660bc6ac9498a979266d8d47c6345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "587a68bb03a8451ea81365b5b05c14fd"
I20260925 14:54:24.327448 18614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72ce782730a4a909489b82297952e8a" candidate_uuid: "842660bc6ac9498a979266d8d47c6345" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a0f68b40c484bbdd3811b2f44441a"
I20260925 14:54:24.327517 18882 raft_consensus.cc:3037] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:24.327540 18614 raft_consensus.cc:3037] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:24.328256 18882 raft_consensus.cc:2454] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842660bc6ac9498a979266d8d47c6345 in term 1.
I20260925 14:54:24.328277 18614 raft_consensus.cc:2454] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 842660bc6ac9498a979266d8d47c6345 in term 1.
I20260925 14:54:24.328471 18683 leader_election.cc:302] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 622a0f68b40c484bbdd3811b2f44441a, 842660bc6ac9498a979266d8d47c6345; no voters:
I20260925 14:54:24.328580 18950 raft_consensus.cc:2781] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:24.328778 18950 raft_consensus.cc:686] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 1 LEADER]: Becoming Leader. State: Replica: 842660bc6ac9498a979266d8d47c6345, State: Running, Role: LEADER
I20260925 14:54:24.328883 18950 consensus_queue.cc:237] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } }
I20260925 14:54:24.329713 18469 catalog_manager.cc:5495] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 reported cstate change: term changed from 0 to 1, leader changed from <none> to 842660bc6ac9498a979266d8d47c6345 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "842660bc6ac9498a979266d8d47c6345" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "842660bc6ac9498a979266d8d47c6345" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:24.365418 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.362853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.98.2:33577)
0925 14:54:24.362924 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.362926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.362935 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:24.362982 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.362983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.363004 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.363065 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:24.363190 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.363660 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.364604 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.364791 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.364847 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.365058 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:24.365147 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:24.365220 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.365290 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.365300 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:54:24.367308 18882 raft_consensus.cc:1261] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 1 FOLLOWER]: Refusing update from remote peer 842660bc6ac9498a979266d8d47c6345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:24.367309 18614 raft_consensus.cc:1261] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 1 FOLLOWER]: Refusing update from remote peer 842660bc6ac9498a979266d8d47c6345: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:24.367636 18950 consensus_queue.cc:1035] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622a0f68b40c484bbdd3811b2f44441a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:24.367815 18956 consensus_queue.cc:1035] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [LEADER]: Connected to new peer: Peer: permanent_uuid: "587a68bb03a8451ea81365b5b05c14fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:24.372259 18469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:24.372375 18469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:24.373077 18469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ba232862faa473aa4f21c07e3ff9af3: Sending DeleteTablet for 3 replicas of tablet a72ce782730a4a909489b82297952e8a
I20260925 14:54:24.373390 18728 tablet_service.cc:1514] Processing DeleteTablet for tablet a72ce782730a4a909489b82297952e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37590
I20260925 14:54:24.373463 18594 tablet_service.cc:1514] Processing DeleteTablet for tablet a72ce782730a4a909489b82297952e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46550
I20260925 14:54:24.373610 18969 tablet_replica.cc:330] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: stopping tablet replica
I20260925 14:54:24.373695 18969 raft_consensus.cc:2229] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:24.373724 18970 tablet_replica.cc:330] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a: stopping tablet replica
I20260925 14:54:24.373708 18862 tablet_service.cc:1514] Processing DeleteTablet for tablet a72ce782730a4a909489b82297952e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43588
I20260925 14:54:24.373790 18969 raft_consensus.cc:2258] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:24.373818 18970 raft_consensus.cc:2229] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:24.373941 18970 raft_consensus.cc:2258] T a72ce782730a4a909489b82297952e8a P 622a0f68b40c484bbdd3811b2f44441a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:24.374114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18531
I20260925 14:54:24.376294 18969 ts_tablet_manager.cc:1905] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:24.378324 18969 ts_tablet_manager.cc:1918] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:24.378424 18969 log.cc:1196] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/a72ce782730a4a909489b82297952e8a
I20260925 14:54:24.378674 18969 ts_tablet_manager.cc:1939] T a72ce782730a4a909489b82297952e8a P 842660bc6ac9498a979266d8d47c6345: Deleting consensus metadata
I20260925 14:54:24.379078 18453 catalog_manager.cc:4841] TS 842660bc6ac9498a979266d8d47c6345 (127.0.98.2:33577): tablet a72ce782730a4a909489b82297952e8a (table test-table [id=16a8fd7f436c4d0da5a712b55bf22300]) successfully deleted
I20260925 14:54:24.379158 18971 tablet_replica.cc:330] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: stopping tablet replica
I20260925 14:54:24.379281 18971 raft_consensus.cc:2229] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:24.379403 18971 raft_consensus.cc:2258] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:24.379848 18971 ts_tablet_manager.cc:1905] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:54:24.380774 18455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33031 (error 108)
I20260925 14:54:24.381083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18665
I20260925 14:54:24.381635 18971 ts_tablet_manager.cc:1918] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:24.381696 18971 log.cc:1196] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/a72ce782730a4a909489b82297952e8a
I20260925 14:54:24.381938 18971 ts_tablet_manager.cc:1939] T a72ce782730a4a909489b82297952e8a P 587a68bb03a8451ea81365b5b05c14fd: Deleting consensus metadata
I20260925 14:54:24.382377 18454 catalog_manager.cc:4841] TS 587a68bb03a8451ea81365b5b05c14fd (127.0.98.3:37635): tablet a72ce782730a4a909489b82297952e8a (table test-table [id=16a8fd7f436c4d0da5a712b55bf22300]) successfully deleted
I20260925 14:54:24.382692 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.382551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33031 (local address 127.0.0.1:46558)
0925 14:54:24.382614 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.382654 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 14:54:24.382966 18455 catalog_manager.cc:4584] TS 622a0f68b40c484bbdd3811b2f44441a (127.0.98.1:33031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a72ce782730a4a909489b82297952e8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33031: connect: Connection refused (error 111)
I20260925 14:54:24.386622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18799
I20260925 14:54:24.391140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18437
2026-09-25T14:54:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2336 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T14:54:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:24Z Disabled control of system clock
I20260925 14:54:24.418200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:24.517964 18976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:24.518193 18976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:24.518213 18976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:24.519481 18976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:24.519526 18976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:24.519537 18976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:24.519550 18976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:24.519593 18976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:24.520931 18976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:24.521193 18976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:24.521484 18976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:24.526505 18982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.526537 18976 server_base.cc:1034] running on GCE node
W20260925 14:54:24.526486 18984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.526482 18981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.527133 18976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:24.527462 18976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:24.528630 18976 hybrid_clock.cc:648] HybridClock initialized: now 1790348064528605 us; error 38 us; skew 500 ppm
I20260925 14:54:24.529995 18976 webserver.cc:460] Webserver started at http://127.0.98.62:33015/ using document root <none> and password file <none>
I20260925 14:54:24.530273 18976 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:24.530324 18976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:24.530522 18976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:24.531669 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e628c657db7946e2858ed826e9cff84e"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "c99da4b3c4e97fce6554d4e1fcb8c55f"
server_key_iv: "1f2f4a812262b4fd7657e2a7e619cfcb"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.532088 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e628c657db7946e2858ed826e9cff84e"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "c99da4b3c4e97fce6554d4e1fcb8c55f"
server_key_iv: "1f2f4a812262b4fd7657e2a7e619cfcb"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.533545 18976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:24.534409 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.534608 18976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:24.534726 18976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e628c657db7946e2858ed826e9cff84e"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "c99da4b3c4e97fce6554d4e1fcb8c55f"
server_key_iv: "1f2f4a812262b4fd7657e2a7e619cfcb"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.534822 18976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:24.543800 18976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:24.544128 18976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:24.544282 18976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:24.548236 18976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41393
I20260925 14:54:24.548353 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41393 every 8 connection(s)
I20260925 14:54:24.548659 18976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:24.549621 19043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:24.551967 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: Bootstrap starting.
I20260925 14:54:24.552505 19043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:24.552738 19043 log.cc:824] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:24.553380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 18976
I20260925 14:54:24.553467 19043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: No bootstrap required, opened a new log
I20260925 14:54:24.553496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:24.553711 392 external_mini_cluster.cc:1428] Setting key e3b78e99eec355e44f7efecbd692ef75
I20260925 14:54:24.555449 19043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 14:54:24.555588 19043 raft_consensus.cc:374] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:24.555608 19043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e628c657db7946e2858ed826e9cff84e, State: Initialized, Role: FOLLOWER
I20260925 14:54:24.555758 19043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 14:54:24.555835 19043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:24.555876 19043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:24.555926 19043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:24.556682 19043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 14:54:24.556794 19043 leader_election.cc:302] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e628c657db7946e2858ed826e9cff84e; no voters:
I20260925 14:54:24.556972 19043 leader_election.cc:288] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:24.557031 19048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:24.557185 19048 raft_consensus.cc:686] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [term 1 LEADER]: Becoming Leader. State: Replica: e628c657db7946e2858ed826e9cff84e, State: Running, Role: LEADER
I20260925 14:54:24.557219 19043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:24.557272 19048 consensus_queue.cc:237] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } }
I20260925 14:54:24.557642 19048 sys_catalog.cc:455] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e628c657db7946e2858ed826e9cff84e. Latest consensus state: current_term: 1 leader_uuid: "e628c657db7946e2858ed826e9cff84e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260925 14:54:24.557720 19048 sys_catalog.cc:458] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:24.557875 19049 sys_catalog.cc:455] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e628c657db7946e2858ed826e9cff84e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e628c657db7946e2858ed826e9cff84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41393 } } }
I20260925 14:54:24.557942 19049 sys_catalog.cc:458] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:24.558065 19055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:24.558776 19055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:24.560590 19055 catalog_manager.cc:1347] Generated new cluster ID: 1519f5bf04c64598af38f404911b80b7
I20260925 14:54:24.560647 19055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:24.561205 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.554265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37720 (local address 127.0.98.62:41393)
0925 14:54:24.554483 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.554490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.554515 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:54:24.555450 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.555460 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.555527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.555721 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.556011 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.557807 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.559210 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.559613 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.559764 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.560504 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:24.560509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:24.560510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:24.560730 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:24.560773 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:24.560787 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:24.560859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.560935 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.561008 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260925 14:54:24.570904 19055 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:24.571434 19055 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:24.573499 19055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: Generated new TSK 0
I20260925 14:54:24.573669 19055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:24.581995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--builtin_ntp_servers=127.0.98.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:24.667112 19067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:24.667318 19067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:24.667340 19067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:24.668507 19067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:24.668548 19067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:24.668574 19067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:24.669947 19067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:24.670194 19067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:24.670526 19067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:24.675500 19075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.675508 19073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.675508 19072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.675952 19067 server_base.cc:1034] running on GCE node
I20260925 14:54:24.676172 19067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:24.676462 19067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:24.677652 19067 hybrid_clock.cc:648] HybridClock initialized: now 1790348064677639 us; error 27 us; skew 500 ppm
I20260925 14:54:24.679090 19067 webserver.cc:460] Webserver started at http://127.0.98.1:34785/ using document root <none> and password file <none>
I20260925 14:54:24.679381 19067 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:24.679435 19067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:24.679592 19067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:24.680706 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "53d22e3020e24466937eebac9ecdd25d"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "d8e0a02e5609a3f3248e8ca67d4f19ae"
server_key_iv: "0eb2191280c2ab034526171aa5732925"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.681103 19067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53d22e3020e24466937eebac9ecdd25d"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "d8e0a02e5609a3f3248e8ca67d4f19ae"
server_key_iv: "0eb2191280c2ab034526171aa5732925"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.682622 19067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:24.683487 19081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.683676 19067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:24.683794 19067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "53d22e3020e24466937eebac9ecdd25d"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "d8e0a02e5609a3f3248e8ca67d4f19ae"
server_key_iv: "0eb2191280c2ab034526171aa5732925"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.683887 19067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:24.703375 19067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:24.703749 19067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:24.703929 19067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:24.704190 19067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:24.704573 19067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:24.704631 19067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.704679 19067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:24.704732 19067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.711238 19067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35879
I20260925 14:54:24.711272 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35879 every 8 connection(s)
I20260925 14:54:24.711649 19067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:24.713553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19067
I20260925 14:54:24.713663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:24.713853 392 external_mini_cluster.cc:1428] Setting key f2ca8a047c2389d90ea4a68c57653384
I20260925 14:54:24.714848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--builtin_ntp_servers=127.0.98.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:54:24.717406 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.712462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54387 (local address 127.0.98.62:41393)
0925 14:54:24.712548 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.712551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.712793 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:54:24.713259 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.713260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.713271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.713325 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.714143 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.714748 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.715917 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.716189 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.716246 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.717162 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:24.717164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:24.717164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:24.717168 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:24.717181 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:24.717185 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:24.717229 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.717324 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.717348 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:54:24.717694 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.712431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.1:54387)
0925 14:54:24.712664 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.712700 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.713174 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.713401 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.713405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.713412 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.714042 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.714048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.714873 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.714874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.715795 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.715797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.716277 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.716278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.716453 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.717011 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:24.717012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.717050 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:24.717078 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:24.717247 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:24.717252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:24.717255 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:24.717333 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:54:24.717442 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260925 14:54:24.718241 19195 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260925 14:54:24.718374 19195 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:24.718554 19195 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260925 14:54:24.719141 19007 ts_manager.cc:194] Registered new tserver with Master: 53d22e3020e24466937eebac9ecdd25d (127.0.98.1:35879)
I20260925 14:54:24.720126 19007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54387
W20260925 14:54:24.796620 19198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:24.796821 19198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:24.796842 19198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:24.797920 19198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:24.797953 19198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:24.797978 19198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:24.799261 19198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:24.799499 19198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:24.799741 19198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:24.804783 19204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.804814 19206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.804757 19203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.804994 19198 server_base.cc:1034] running on GCE node
I20260925 14:54:24.805277 19198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:24.805555 19198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:24.806726 19198 hybrid_clock.cc:648] HybridClock initialized: now 1790348064806706 us; error 29 us; skew 500 ppm
I20260925 14:54:24.808046 19198 webserver.cc:460] Webserver started at http://127.0.98.2:38909/ using document root <none> and password file <none>
I20260925 14:54:24.808341 19198 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:24.808415 19198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:24.808561 19198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:24.809623 19198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7c49d05a2e04ea490c92352408dfb15"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "bc4b348ed8c1bf3b0aeedb49496f44ac"
server_key_iv: "9a9b40c1d4dea9a96b236dd09b1bce52"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.809986 19198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7c49d05a2e04ea490c92352408dfb15"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "bc4b348ed8c1bf3b0aeedb49496f44ac"
server_key_iv: "9a9b40c1d4dea9a96b236dd09b1bce52"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.811460 19198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:24.812301 19212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.812502 19198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:24.812597 19198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "d7c49d05a2e04ea490c92352408dfb15"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "bc4b348ed8c1bf3b0aeedb49496f44ac"
server_key_iv: "9a9b40c1d4dea9a96b236dd09b1bce52"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.812677 19198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:24.843465 19198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:24.843828 19198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:24.844000 19198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:24.844265 19198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:24.844624 19198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:24.844682 19198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.844723 19198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:24.844765 19198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.850383 19198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40329
I20260925 14:54:24.850464 19325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40329 every 8 connection(s)
I20260925 14:54:24.850827 19198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:24.855852 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.851585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36441 (local address 127.0.98.62:41393)
0925 14:54:24.851657 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.851660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.851928 (+ 268us) server_negotiation.cc:407] Connection header received
0925 14:54:24.852393 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.852394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.852404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.852456 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.853133 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.853626 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.854633 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.854821 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.854873 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.855626 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:24.855627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:24.855627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:24.855631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:24.855645 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:24.855649 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:24.855691 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.855770 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.855793 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:24.856058 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.851578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.2:36441)
0925 14:54:24.851793 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.851828 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.852308 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.852526 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.852529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.852538 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.853053 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.853058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.853735 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.853736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.854558 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.854560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.854894 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.854895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.854999 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.855497 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:24.855498 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.855526 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:24.855549 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:24.855702 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:24.855705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:24.855708 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:24.855759 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:54:24.855843 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260925 14:54:24.856597 19326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260925 14:54:24.856709 19326 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:24.856920 19326 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260925 14:54:24.857332 19007 ts_manager.cc:194] Registered new tserver with Master: d7c49d05a2e04ea490c92352408dfb15 (127.0.98.2:40329)
I20260925 14:54:24.857752 19007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36441
I20260925 14:54:24.860350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19198
I20260925 14:54:24.860430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:24.860580 392 external_mini_cluster.cc:1428] Setting key 96611ea4f2eb951120c4f16363456e86
I20260925 14:54:24.861431 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--builtin_ntp_servers=127.0.98.20:34555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:24.941301 19329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:24.941511 19329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:24.941533 19329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:24.943040 19329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:24.943089 19329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:24.943115 19329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:24.944621 19329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:24.944877 19329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:24.945176 19329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:24.950241 19335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.950373 19329 server_base.cc:1034] running on GCE node
W20260925 14:54:24.950215 19334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:24.950215 19337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:24.950727 19329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:24.950932 19329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:24.952065 19329 hybrid_clock.cc:648] HybridClock initialized: now 1790348064952054 us; error 26 us; skew 500 ppm
I20260925 14:54:24.953217 19329 webserver.cc:460] Webserver started at http://127.0.98.3:40693/ using document root <none> and password file <none>
I20260925 14:54:24.953477 19329 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:24.953524 19329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:24.953693 19329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:24.954846 19329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "c61ebce105fa47c993c0cd3e3a63ac04"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "f4d7e90920012bfe5bf636be589f2b72"
server_key_iv: "5f9ad38338c2e16fca8f48690e620c3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.955195 19329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c61ebce105fa47c993c0cd3e3a63ac04"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "f4d7e90920012bfe5bf636be589f2b72"
server_key_iv: "5f9ad38338c2e16fca8f48690e620c3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.956480 19329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:24.957247 19343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.957430 19329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:24.957530 19329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "c61ebce105fa47c993c0cd3e3a63ac04"
format_stamp: "Formatted at 2026-09-25 14:54:24 on dist-test-slave-gpc1"
server_key: "f4d7e90920012bfe5bf636be589f2b72"
server_key_iv: "5f9ad38338c2e16fca8f48690e620c3e"
server_key_version: "encryptionkey@0"
I20260925 14:54:24.957598 19329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:24.964659 19329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:24.964972 19329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:24.965072 19329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:24.965238 19329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:24.965560 19329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:24.965618 19329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.965658 19329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:24.965685 19329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:24.971377 19329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44705
I20260925 14:54:24.971458 19456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44705 every 8 connection(s)
I20260925 14:54:24.971767 19329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:24.975605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19329
I20260925 14:54:24.975719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:24.975929 392 external_mini_cluster.cc:1428] Setting key defdc3230a2b01d471dc1c9472b50158
I20260925 14:54:24.977156 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.972523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44127 (local address 127.0.98.62:41393)
0925 14:54:24.972570 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.972572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.972816 (+ 244us) server_negotiation.cc:407] Connection header received
0925 14:54:24.973348 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.973350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.973363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.973415 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.974084 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.974651 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.975629 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.975916 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.975993 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.976886 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:24.976887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:24.976888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:24.976892 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:24.976908 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:24.976912 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:24.976959 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.977051 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.977081 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:24.977372 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.972505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41393 (local address 127.0.98.3:44127)
0925 14:54:24.972706 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.972736 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.973235 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.973482 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.973486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.973496 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:24.973990 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.973995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.974752 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.974753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.975541 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.975543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.976001 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.976002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:24.976139 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.976732 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:24.976733 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:24.976761 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:24.976782 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:24.976961 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:24.976966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:24.976969 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:24.977054 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 14:54:24.977152 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260925 14:54:24.977798 19457 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41393
I20260925 14:54:24.977886 19457 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:24.978060 19457 heartbeater.cc:511] Master 127.0.98.62:41393 requested a full tablet report, sending...
I20260925 14:54:24.978438 19007 ts_manager.cc:194] Registered new tserver with Master: c61ebce105fa47c993c0cd3e3a63ac04 (127.0.98.3:44705)
I20260925 14:54:24.978792 19007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44127
I20260925 14:54:24.987586 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:24.991767 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.989000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.0.98.62:41393)
0925 14:54:24.989056 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.989058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.989116 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:54:24.989162 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.989162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.989172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.989228 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:24.989380 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.989832 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.990681 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.990863 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.990918 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:24.991547 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:24.991548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:24.991548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:24.991552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:24.991565 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:24.991568 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:24.991640 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:24.991706 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:54:24.991715 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:54:24.993192 19007 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:24.994861 19007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:25.001085 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.997309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.98.2:40329)
0925 14:54:24.997532 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.997535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.997701 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:54:24.997742 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.997745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.997761 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.997831 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:24.998011 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.999100 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000148 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.000665 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000752 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.000851 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.000947 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.000987 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":91,"threads_started":1}
I20260925 14:54:25.001068 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.997206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.0.1:56504)
0925 14:54:24.997572 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.997579 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.997645 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.997900 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.997901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.997903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:24.997936 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.997937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.999176 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.999177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.000063 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.000065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.000751 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.000838 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.000851 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:54:25.000928 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:54:25.000987 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":65,"threads_started":1}
I20260925 14:54:25.001248 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.996979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35879 (local address 127.0.0.1:55946)
0925 14:54:24.997123 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.997168 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.997234 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.997755 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.997756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.997759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:24.997803 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.997805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.998787 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.998788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.000067 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.000069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.001024 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.001025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.001086 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.001090 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:25.001123 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 14:54:25.001168 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 14:54:25.001421 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.996979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44705 (local address 127.0.0.1:46068)
0925 14:54:24.997326 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:24.997336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:24.997387 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:24.997668 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:24.997671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:24.997673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:24.997740 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:24.997742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.999073 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:24.999074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.000102 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.000104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.001218 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.001219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.001278 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.001281 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:25.001325 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:54:25.001365 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260925 14:54:25.001618 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.997286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.0.98.1:35879)
0925 14:54:24.997465 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.997471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.997481 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:24.997523 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.997525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.997542 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.997885 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:24.997961 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.998705 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000176 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.000911 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.001354 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.001462 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.001490 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.001533 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260925 14:54:25.001729 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:24.997090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.0.98.3:44705)
0925 14:54:24.997489 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:24.997493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:24.997503 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:24.997536 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:24.997538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:24.997551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:24.997633 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:24.997847 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:24.998988 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000190 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.000911 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.000984 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.001564 (+ 580us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.001589 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.001635 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":98,"threads_started":1}
I20260925 14:54:25.001942 19260 tablet_service.cc:1467] Processing CreateTablet for tablet 66890493a0bc49eba3d7b174028aeb31 (DEFAULT_TABLE table=test-table [id=03bf5ad4b154485286a61cd7ba0db912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:25.002336 19260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66890493a0bc49eba3d7b174028aeb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:25.002383 19129 tablet_service.cc:1467] Processing CreateTablet for tablet 66890493a0bc49eba3d7b174028aeb31 (DEFAULT_TABLE table=test-table [id=03bf5ad4b154485286a61cd7ba0db912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:25.002745 19129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66890493a0bc49eba3d7b174028aeb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:25.003026 19391 tablet_service.cc:1467] Processing CreateTablet for tablet 66890493a0bc49eba3d7b174028aeb31 (DEFAULT_TABLE table=test-table [id=03bf5ad4b154485286a61cd7ba0db912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:25.003335 19391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66890493a0bc49eba3d7b174028aeb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:25.005363 19471 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: Bootstrap starting.
I20260925 14:54:25.005790 19472 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Bootstrap starting.
I20260925 14:54:25.006157 19471 tablet_bootstrap.cc:654] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:25.006466 19472 tablet_bootstrap.cc:654] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:25.006477 19473 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Bootstrap starting.
I20260925 14:54:25.006522 19471 log.cc:824] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:25.006773 19472 log.cc:824] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:25.007150 19473 tablet_bootstrap.cc:654] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:25.007489 19473 log.cc:824] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:25.007536 19472 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: No bootstrap required, opened a new log
I20260925 14:54:25.007666 19472 ts_tablet_manager.cc:1397] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:54:25.007825 19471 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: No bootstrap required, opened a new log
I20260925 14:54:25.007889 19471 ts_tablet_manager.cc:1397] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:54:25.008183 19473 tablet_bootstrap.cc:492] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: No bootstrap required, opened a new log
I20260925 14:54:25.008262 19473 ts_tablet_manager.cc:1397] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:25.009977 19473 raft_consensus.cc:348] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010133 19473 raft_consensus.cc:374] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:25.010176 19473 raft_consensus.cc:729] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c61ebce105fa47c993c0cd3e3a63ac04, State: Initialized, Role: FOLLOWER
I20260925 14:54:25.010306 19473 consensus_queue.cc:260] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010288 19472 raft_consensus.cc:348] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010314 19471 raft_consensus.cc:348] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010473 19471 raft_consensus.cc:374] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:25.010466 19472 raft_consensus.cc:374] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:25.010505 19471 raft_consensus.cc:729] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53d22e3020e24466937eebac9ecdd25d, State: Initialized, Role: FOLLOWER
I20260925 14:54:25.010505 19472 raft_consensus.cc:729] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c49d05a2e04ea490c92352408dfb15, State: Initialized, Role: FOLLOWER
I20260925 14:54:25.010619 19473 ts_tablet_manager.cc:1428] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:54:25.010608 19471 consensus_queue.cc:260] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010628 19472 consensus_queue.cc:260] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.010756 19457 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260925 14:54:25.010838 19471 ts_tablet_manager.cc:1428] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:25.010895 19195 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260925 14:54:25.011137 19472 ts_tablet_manager.cc:1428] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:54:25.011340 19326 heartbeater.cc:503] Master 127.0.98.62:41393 was elected leader, sending a full tablet report...
I20260925 14:54:25.078949 19478 raft_consensus.cc:482] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:25.079097 19478 raft_consensus.cc:504] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.079509 19478 leader_election.cc:288] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c61ebce105fa47c993c0cd3e3a63ac04 (127.0.98.3:44705), d7c49d05a2e04ea490c92352408dfb15 (127.0.98.2:40329)
I20260925 14:54:25.082247 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.079656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44705 (local address 127.0.98.1:33553)
0925 14:54:25.079712 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.079725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:25.079773 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:25.079944 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:25.079945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:25.079947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:25.080017 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.080018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.080652 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.080653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.081396 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.081398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.081986 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.081987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.082064 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.082067 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:25.082127 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:54:25.082170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260925 14:54:25.082247 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.079721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33553 (local address 127.0.98.3:44705)
0925 14:54:25.079783 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.079785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.079798 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:54:25.079832 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.079833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.079842 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.079920 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:25.080087 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.080566 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.081502 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.081883 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.081986 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.081998 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.082145 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.082184 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:54:25.082397 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.079706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40329 (local address 127.0.98.1:47145)
0925 14:54:25.079870 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.079879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:25.079917 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:25.080061 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:25.080062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:25.080064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:25.080137 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.080138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.080753 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.080754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.081496 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:25.081498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.082083 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.082084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:25.082162 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.082165 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:25.082240 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:54:25.082280 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":104,"threads_started":1}
I20260925 14:54:25.082572 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.079877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47145 (local address 127.0.98.2:40329)
0925 14:54:25.079925 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.079927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.079938 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:25.079979 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.079980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.079990 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.080057 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:25.080208 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.080669 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.081608 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.081985 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.082082 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.082091 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.082249 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.082286 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:25.082638 19411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66890493a0bc49eba3d7b174028aeb31" candidate_uuid: "53d22e3020e24466937eebac9ecdd25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" is_pre_election: true
I20260925 14:54:25.082809 19411 raft_consensus.cc:2454] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d22e3020e24466937eebac9ecdd25d in term 0.
I20260925 14:54:25.082854 19280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66890493a0bc49eba3d7b174028aeb31" candidate_uuid: "53d22e3020e24466937eebac9ecdd25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c49d05a2e04ea490c92352408dfb15" is_pre_election: true
I20260925 14:54:25.083024 19280 raft_consensus.cc:2454] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53d22e3020e24466937eebac9ecdd25d in term 0.
I20260925 14:54:25.083060 19085 leader_election.cc:302] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53d22e3020e24466937eebac9ecdd25d, c61ebce105fa47c993c0cd3e3a63ac04; no voters:
I20260925 14:54:25.083176 19478 raft_consensus.cc:2781] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:25.083278 19478 raft_consensus.cc:482] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:25.083315 19478 raft_consensus.cc:3037] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:25.084005 19478 raft_consensus.cc:504] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.084172 19478 leader_election.cc:288] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [CANDIDATE]: Term 1 election: Requested vote from peers c61ebce105fa47c993c0cd3e3a63ac04 (127.0.98.3:44705), d7c49d05a2e04ea490c92352408dfb15 (127.0.98.2:40329)
I20260925 14:54:25.084314 19411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66890493a0bc49eba3d7b174028aeb31" candidate_uuid: "53d22e3020e24466937eebac9ecdd25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c61ebce105fa47c993c0cd3e3a63ac04"
I20260925 14:54:25.084331 19280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66890493a0bc49eba3d7b174028aeb31" candidate_uuid: "53d22e3020e24466937eebac9ecdd25d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c49d05a2e04ea490c92352408dfb15"
I20260925 14:54:25.084414 19411 raft_consensus.cc:3037] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:25.084438 19280 raft_consensus.cc:3037] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:25.085150 19280 raft_consensus.cc:2454] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d22e3020e24466937eebac9ecdd25d in term 1.
I20260925 14:54:25.085151 19411 raft_consensus.cc:2454] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53d22e3020e24466937eebac9ecdd25d in term 1.
I20260925 14:54:25.085330 19082 leader_election.cc:302] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53d22e3020e24466937eebac9ecdd25d, d7c49d05a2e04ea490c92352408dfb15; no voters:
I20260925 14:54:25.085429 19478 raft_consensus.cc:2781] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:25.085562 19478 raft_consensus.cc:686] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [term 1 LEADER]: Becoming Leader. State: Replica: 53d22e3020e24466937eebac9ecdd25d, State: Running, Role: LEADER
I20260925 14:54:25.085664 19478 consensus_queue.cc:237] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } }
I20260925 14:54:25.086567 19006 catalog_manager.cc:5495] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d reported cstate change: term changed from 0 to 1, leader changed from <none> to 53d22e3020e24466937eebac9ecdd25d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "53d22e3020e24466937eebac9ecdd25d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53d22e3020e24466937eebac9ecdd25d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:25.094975 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.092580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55954 (local address 127.0.98.1:35879)
0925 14:54:25.092642 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:25.092644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:25.092654 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:54:25.092720 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:25.092721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:25.092742 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:25.092818 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:25.092960 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.093396 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.094154 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:25.094377 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:25.094457 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:25.094622 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:25.094724 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:25.094831 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:25.094861 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:54:25.094871 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:25.096585 19280 raft_consensus.cc:1261] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 1 FOLLOWER]: Refusing update from remote peer 53d22e3020e24466937eebac9ecdd25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:25.096640 19411 raft_consensus.cc:1261] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 1 FOLLOWER]: Refusing update from remote peer 53d22e3020e24466937eebac9ecdd25d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:25.096921 19481 consensus_queue.cc:1035] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c49d05a2e04ea490c92352408dfb15" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:25.097054 19478 consensus_queue.cc:1035] T 66890493a0bc49eba3d7b174028aeb31 P 53d22e3020e24466937eebac9ecdd25d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c61ebce105fa47c993c0cd3e3a63ac04" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:25.099493 19486 mvcc.cc:205] Tried to move back new op lower bound from 7333265674633244672 to 7333265674591625216. Current Snapshot: MvccSnapshot[applied={T|T < 7333265674633244672}]
I20260925 14:54:25.099566 19485 mvcc.cc:205] Tried to move back new op lower bound from 7333265674633244672 to 7333265674591625216. Current Snapshot: MvccSnapshot[applied={T|T < 7333265674633244672}]
I20260925 14:54:25.101459 19007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:25.101553 19007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:25.102269 19007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e628c657db7946e2858ed826e9cff84e: Sending DeleteTablet for 3 replicas of tablet 66890493a0bc49eba3d7b174028aeb31
I20260925 14:54:25.102654 19260 tablet_service.cc:1514] Processing DeleteTablet for tablet 66890493a0bc49eba3d7b174028aeb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:25 UTC) from {username='slave'} at 127.0.0.1:56504
I20260925 14:54:25.102643 19391 tablet_service.cc:1514] Processing DeleteTablet for tablet 66890493a0bc49eba3d7b174028aeb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:25 UTC) from {username='slave'} at 127.0.0.1:46068
I20260925 14:54:25.102860 19494 tablet_replica.cc:330] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: stopping tablet replica
I20260925 14:54:25.102958 19494 raft_consensus.cc:2229] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:25.102934 19129 tablet_service.cc:1514] Processing DeleteTablet for tablet 66890493a0bc49eba3d7b174028aeb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:25 UTC) from {username='slave'} at 127.0.0.1:55946
I20260925 14:54:25.103044 19494 raft_consensus.cc:2258] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:25.103050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19067
I20260925 14:54:25.104135 19494 ts_tablet_manager.cc:1905] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:25.105535 19494 ts_tablet_manager.cc:1918] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:25.105584 19494 log.cc:1196] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/66890493a0bc49eba3d7b174028aeb31
I20260925 14:54:25.105856 19494 ts_tablet_manager.cc:1939] T 66890493a0bc49eba3d7b174028aeb31 P d7c49d05a2e04ea490c92352408dfb15: Deleting consensus metadata
I20260925 14:54:25.106308 18991 catalog_manager.cc:4841] TS d7c49d05a2e04ea490c92352408dfb15 (127.0.98.2:40329): tablet 66890493a0bc49eba3d7b174028aeb31 (table test-table [id=03bf5ad4b154485286a61cd7ba0db912]) successfully deleted
I20260925 14:54:25.109242 19495 tablet_replica.cc:330] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: stopping tablet replica
W20260925 14:54:25.109241 18991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35879 (error 108)
I20260925 14:54:25.109349 19495 raft_consensus.cc:2229] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:25.109414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19198
I20260925 14:54:25.109454 19495 raft_consensus.cc:2258] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:25.110414 19495 ts_tablet_manager.cc:1905] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:25.111673 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:25.111527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35879 (local address 127.0.0.1:55968)
0925 14:54:25.111588 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:25.111632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 14:54:25.111794 18991 catalog_manager.cc:4584] TS 53d22e3020e24466937eebac9ecdd25d (127.0.98.1:35879): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66890493a0bc49eba3d7b174028aeb31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35879: connect: Connection refused (error 111)
I20260925 14:54:25.112339 19495 ts_tablet_manager.cc:1918] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:25.112417 19495 log.cc:1196] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/66890493a0bc49eba3d7b174028aeb31
I20260925 14:54:25.112651 19495 ts_tablet_manager.cc:1939] T 66890493a0bc49eba3d7b174028aeb31 P c61ebce105fa47c993c0cd3e3a63ac04: Deleting consensus metadata
I20260925 14:54:25.113214 18994 catalog_manager.cc:4841] TS c61ebce105fa47c993c0cd3e3a63ac04 (127.0.98.3:44705): tablet 66890493a0bc49eba3d7b174028aeb31 (table test-table [id=03bf5ad4b154485286a61cd7ba0db912]) successfully deleted
I20260925 14:54:25.114409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19329
I20260925 14:54:25.119274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 18976
2026-09-25T14:54:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (727 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:25 dist-test-slave-gpc1 krb5kdc[19500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:25 dist-test-slave-gpc1 krb5kdc[19500](info): set up 2 sockets
Sep 25 14:54:25 dist-test-slave-gpc1 krb5kdc[19500](info): commencing operation
krb5kdc: starting...
W20260925 14:54:26.158198 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:26.273293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:26.353704 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:26.353958 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:26.353994 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:26.355190 19513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:26.355274 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:26.355299 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:26.355326 19513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:26.355358 19513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:26.356812 19513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:26.357074 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:26.357291 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:26.362241 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.362178 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:26.362216 19513 server_base.cc:1034] running on GCE node
W20260925 14:54:26.362177 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:26.362740 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:26.363054 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:26.364241 19513 hybrid_clock.cc:648] HybridClock initialized: now 1790348066364223 us; error 48 us; skew 500 ppm
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:26.366762 19513 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:26.367182 19513 webserver.cc:460] Webserver started at http://127.0.98.62:41433/ using document root <none> and password file <none>
I20260925 14:54:26.367408 19513 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:26.367475 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:26.367640 19513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:26.368681 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "49d83ca6a976422db53d4e243965bed4"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "60f00ff9cdab6678330267230771017d"
server_key_iv: "d1d0299c967ac19ccacbaeca87e609f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.369068 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "49d83ca6a976422db53d4e243965bed4"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "60f00ff9cdab6678330267230771017d"
server_key_iv: "d1d0299c967ac19ccacbaeca87e609f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.370421 19513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:54:26.371136 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.371326 19513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:26.371439 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "49d83ca6a976422db53d4e243965bed4"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "60f00ff9cdab6678330267230771017d"
server_key_iv: "d1d0299c967ac19ccacbaeca87e609f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.371512 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:26.390666 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:26.392521 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:26.392688 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:26.396606 19513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44675
I20260925 14:54:26.396642 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44675 every 8 connection(s)
I20260925 14:54:26.397006 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:26.397886 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:26.398137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19513
I20260925 14:54:26.398249 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:26.398505 392 external_mini_cluster.cc:1428] Setting key 4ada25d3e7814c5219284d092d5b2b57
I20260925 14:54:26.400611 19581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: Bootstrap starting.
I20260925 14:54:26.401199 19581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:26.401501 19581 log.cc:824] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:26.402292 19581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: No bootstrap required, opened a new log
I20260925 14:54:26.404815 19581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260925 14:54:26.405019 19581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:26.405059 19581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d83ca6a976422db53d4e243965bed4, State: Initialized, Role: FOLLOWER
I20260925 14:54:26.405201 19581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260925 14:54:26.405277 19581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:26.405299 19581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:26.405365 19581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:26.405965 19581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260925 14:54:26.406080 19581 leader_election.cc:302] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49d83ca6a976422db53d4e243965bed4; no voters:
I20260925 14:54:26.406257 19581 leader_election.cc:288] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:26.406291 19586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:26.406436 19586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [term 1 LEADER]: Becoming Leader. State: Replica: 49d83ca6a976422db53d4e243965bed4, State: Running, Role: LEADER
I20260925 14:54:26.406546 19581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:26.406585 19586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } }
I20260925 14:54:26.406913 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49d83ca6a976422db53d4e243965bed4. Latest consensus state: current_term: 1 leader_uuid: "49d83ca6a976422db53d4e243965bed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } } }
I20260925 14:54:26.407032 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:26.406929 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49d83ca6a976422db53d4e243965bed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49d83ca6a976422db53d4e243965bed4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44675 } } }
I20260925 14:54:26.407349 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:26.407239 19592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:26.407903 19592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:26.409436 19592 catalog_manager.cc:1347] Generated new cluster ID: 255ea752427947988a7ed23d9971950e
I20260925 14:54:26.409487 19592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:26.410435 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.399085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54078 (local address 127.0.98.62:44675)
0925 14:54:26.399283 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.399300 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.399329 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:54:26.399747 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.399754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.399799 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.399974 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.400674 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.401710 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.402779 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.403199 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.403282 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.406159 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:26.406163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:26.406164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:26.406198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:26.408366 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.408746 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.408749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.408750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.408878 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.409133 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.409134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.409134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.409446 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:26.409973 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.410055 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.410195 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260925 14:54:26.420997 19592 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:26.421747 19592 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:26.429280 19592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: Generated new TSK 0
I20260925 14:54:26.429492 19592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:54:26.457923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:26.542840 19607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:26.543066 19607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:26.543105 19607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:26.544291 19607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:26.544356 19607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:26.544402 19607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:26.545913 19607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:26.546172 19607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:26.546463 19607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:26.551386 19615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.551386 19612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.551585 19613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:26.551797 19607 server_base.cc:1034] running on GCE node
I20260925 14:54:26.552032 19607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:26.552279 19607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:26.553442 19607 hybrid_clock.cc:648] HybridClock initialized: now 1790348066553405 us; error 53 us; skew 500 ppm
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:26.556005 19607 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:54:26.556438 19607 webserver.cc:460] Webserver started at http://127.0.98.1:32833/ using document root <none> and password file <none>
I20260925 14:54:26.556675 19607 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:26.556751 19607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:26.556896 19607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:26.557981 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b3560ec5f134c7194b58f028fdf11f3"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "7bae36a8251d94e14cccc2837a16b5ba"
server_key_iv: "18e0e0e952b717c3633c64331723c284"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.558454 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b3560ec5f134c7194b58f028fdf11f3"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "7bae36a8251d94e14cccc2837a16b5ba"
server_key_iv: "18e0e0e952b717c3633c64331723c284"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.559769 19607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:26.560519 19622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.560865 19607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:26.560971 19607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "1b3560ec5f134c7194b58f028fdf11f3"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "7bae36a8251d94e14cccc2837a16b5ba"
server_key_iv: "18e0e0e952b717c3633c64331723c284"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.561059 19607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:26.579960 19607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:26.581938 19607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:26.582123 19607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:26.582468 19607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:26.582827 19607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:26.582886 19607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.582932 19607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:26.582966 19607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.589447 19607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41911
I20260925 14:54:26.589498 19735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41911 every 8 connection(s)
I20260925 14:54:26.589859 19607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:26.593113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19607
I20260925 14:54:26.593243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:26.593500 392 external_mini_cluster.cc:1428] Setting key 51841c820f37becb66e6e8a9503c9f90
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:26.600425 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.590612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47827 (local address 127.0.98.62:44675)
0925 14:54:26.590682 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.590685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.590893 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:54:26.591897 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.591899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.591913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.591974 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.592659 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.593189 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.594768 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.594967 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.595025 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.597349 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:26.597351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:26.597351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:26.597358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:26.598895 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.599420 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.599422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.599422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.599481 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.599792 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.599793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.599793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.599896 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:26.599962 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.600279 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.600351 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:54:26.600677 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.590581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.1:47827)
0925 14:54:26.590750 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.590802 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.591773 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.592047 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.592051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.592342 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.592540 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.592545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.593322 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.593323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.594659 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.594662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.595052 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.595053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.595202 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.595815 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.595823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.597259 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:26.598975 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.598983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.598986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.599322 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.599567 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.599568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.599569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.599720 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.599961 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:26.599968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:26.600135 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.600262 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.600435 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260925 14:54:26.601193 19736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260925 14:54:26.601279 19736 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:26.601480 19736 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260925 14:54:26.602041 19545 ts_manager.cc:194] Registered new tserver with Master: 1b3560ec5f134c7194b58f028fdf11f3 (127.0.98.1:41911)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:54:26.602761 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:54:26.620694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:26.703073 19741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:26.703323 19741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:26.703363 19741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:26.704550 19741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:26.704622 19741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:26.704666 19741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:26.706208 19741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:26.706501 19741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:26.706773 19741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:26.711807 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.711861 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.711835 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:26.712141 19741 server_base.cc:1034] running on GCE node
I20260925 14:54:26.712343 19741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:26.712587 19741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:26.713727 19741 hybrid_clock.cc:648] HybridClock initialized: now 1790348066713702 us; error 38 us; skew 500 ppm
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:26.716259 19741 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:54:26.716727 19741 webserver.cc:460] Webserver started at http://127.0.98.2:40161/ using document root <none> and password file <none>
I20260925 14:54:26.716948 19741 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:26.717012 19741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:26.717154 19741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:26.718178 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "cab987c2bc1d4b3fa2c55273d584b2fa"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "8af9fb8e071191a899b248586e8305a2"
server_key_iv: "e06da4c070ba8bd9a4aff47c30dbd135"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.718568 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cab987c2bc1d4b3fa2c55273d584b2fa"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "8af9fb8e071191a899b248586e8305a2"
server_key_iv: "e06da4c070ba8bd9a4aff47c30dbd135"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.719852 19741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:26.720530 19756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.720696 19741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:26.720793 19741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "cab987c2bc1d4b3fa2c55273d584b2fa"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "8af9fb8e071191a899b248586e8305a2"
server_key_iv: "e06da4c070ba8bd9a4aff47c30dbd135"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.720867 19741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:26.755919 19741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:26.757983 19741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:26.758144 19741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:26.758431 19741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:26.758821 19741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:26.758877 19741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.758920 19741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:26.758947 19741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.764871 19741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37611
I20260925 14:54:26.764979 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37611 every 8 connection(s)
I20260925 14:54:26.765272 19741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:26.765702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19741
I20260925 14:54:26.765825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:26.766016 392 external_mini_cluster.cc:1428] Setting key a0d3d1a42d3bbb82b398627244a92f88
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:54:26.777155 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.766396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37377 (local address 127.0.98.62:44675)
0925 14:54:26.766436 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.766438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.766447 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:26.767710 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.767741 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.767754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.767819 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.768746 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.769543 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.771189 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.771387 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.771451 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.774056 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:26.774058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:26.774058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:26.774067 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:26.775874 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.776250 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.776251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.776251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.776326 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.776598 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.776599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.776599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.776731 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:26.776787 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.777029 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.777090 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:26.777382 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.765966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.2:37377)
0925 14:54:26.766175 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.766250 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.767557 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.767923 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.767927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.768338 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.768641 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.768646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.769663 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.769665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.771104 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.771107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.771479 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.771481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.771634 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.772211 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.772223 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.773940 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:26.775962 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.775968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.775970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.776187 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.776412 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.776413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.776414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.776531 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.776790 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:26.776794 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:26.776912 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.776995 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.777117 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 14:54:26.778050 19870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260925 14:54:26.778160 19870 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:26.778388 19870 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260925 14:54:26.778807 19545 ts_manager.cc:194] Registered new tserver with Master: cab987c2bc1d4b3fa2c55273d584b2fa (127.0.98.2:37611)
I20260925 14:54:26.779189 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:54:26.792330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:26.872538 19875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:26.872741 19875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:26.872766 19875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:26.873927 19875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:26.873970 19875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:26.873996 19875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:26.875521 19875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:26.875710 19875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:26.875886 19875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:26.880769 19880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.880789 19881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:26.880780 19883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:26.881001 19875 server_base.cc:1034] running on GCE node
I20260925 14:54:26.881263 19875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:26.881505 19875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:26.882673 19875 hybrid_clock.cc:648] HybridClock initialized: now 1790348066882668 us; error 41 us; skew 500 ppm
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:26.885272 19875 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:54:26.885700 19875 webserver.cc:460] Webserver started at http://127.0.98.3:43035/ using document root <none> and password file <none>
I20260925 14:54:26.885928 19875 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:26.885995 19875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:26.886147 19875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:26.887296 19875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "c66299b117ea41a880c16e53ba7ed083"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "71ae07b71ba875e53df3da233360fd9b"
server_key_iv: "65244f192b88bafa55e219cb546606d0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.887661 19875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c66299b117ea41a880c16e53ba7ed083"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "71ae07b71ba875e53df3da233360fd9b"
server_key_iv: "65244f192b88bafa55e219cb546606d0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.888944 19875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:26.889642 19890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.889823 19875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:26.889920 19875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "c66299b117ea41a880c16e53ba7ed083"
format_stamp: "Formatted at 2026-09-25 14:54:26 on dist-test-slave-gpc1"
server_key: "71ae07b71ba875e53df3da233360fd9b"
server_key_iv: "65244f192b88bafa55e219cb546606d0"
server_key_version: "encryptionkey@0"
I20260925 14:54:26.890005 19875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:26.920732 19875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:26.923220 19875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:26.923388 19875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:26.923707 19875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:26.924033 19875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:26.924088 19875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.924150 19875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:26.924180 19875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:26.930392 19875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41345
I20260925 14:54:26.930451 20003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41345 every 8 connection(s)
I20260925 14:54:26.930802 19875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:54:26 dist-test-slave-gpc1 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:26.937003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 19875
I20260925 14:54:26.937103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:26.937479 392 external_mini_cluster.cc:1428] Setting key 5b842d9d31825fcf17d9f009194ad7b1
I20260925 14:54:26.940099 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.931526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57545 (local address 127.0.98.62:44675)
0925 14:54:26.931588 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.931590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.931792 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:54:26.932685 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.932686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.932695 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.932774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.933344 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.933831 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.934933 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.935138 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.935206 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.937351 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:26.937353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:26.937353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:26.937360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:26.938967 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.939351 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.939352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.939352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.939408 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.939626 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.939627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.939627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.939727 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:26.939777 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.939975 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.940038 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:54:26.940291 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.931497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44675 (local address 127.0.98.3:57545)
0925 14:54:26.931658 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.931699 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.932601 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.932834 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.932838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.933083 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:26.933276 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.933280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.933914 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.933915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.934842 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.934844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.935215 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.935216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.935312 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.935786 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:26.935794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:26.937177 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:26.939036 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.939042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.939044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.939263 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.939479 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:26.939480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:26.939480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:26.939573 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:26.939767 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:26.939770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:26.939876 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.939950 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.940061 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 14:54:26.940752 20004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44675
I20260925 14:54:26.940840 20004 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:26.940984 20004 heartbeater.cc:511] Master 127.0.98.62:44675 requested a full tablet report, sending...
I20260925 14:54:26.941308 19545 ts_manager.cc:194] Registered new tserver with Master: c66299b117ea41a880c16e53ba7ed083 (127.0.98.3:41345)
I20260925 14:54:26.941635 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57545
I20260925 14:54:26.949283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:26.956355 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.950583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.0.98.62:44675)
0925 14:54:26.950625 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.950627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.950712 (+ 85us) server_negotiation.cc:407] Connection header received
0925 14:54:26.950761 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.950762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.950770 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.950815 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:26.951332 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.951798 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.952501 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.952677 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.952737 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.953887 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:26.953889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:26.953889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:26.953895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:26.955337 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.955640 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.955641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.955642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.955690 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:26.955853 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:26.955853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:26.955853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:26.955987 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:26.956062 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.956140 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.956196 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:26.958009 19545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:26.959704 19545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:26.966529 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.1:41911)
0925 14:54:26.962538 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.962553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.962562 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:26.962596 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.962598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.962613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.962917 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:26.963030 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.964156 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.965342 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.966091 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.966170 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.966231 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.966412 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.966452 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":80,"threads_started":1}
I20260925 14:54:26.966540 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41911 (local address 127.0.0.1:35526)
0925 14:54:26.962238 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.962278 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.962382 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.962776 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.962777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.962779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:26.962836 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.962838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.964276 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.964277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.965252 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.965255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.966181 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.966182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.966277 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.966291 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.966385 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.966450 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260925 14:54:26.966786 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.0.98.3:41345)
0925 14:54:26.962613 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.962619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.962753 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:54:26.962793 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.962796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.962811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.962883 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:26.963077 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.963870 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.965447 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.966104 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.966170 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.966231 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.966668 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.966709 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260925 14:54:26.966540 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41345 (local address 127.0.0.1:49572)
0925 14:54:26.962631 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.962641 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.962697 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.962975 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.962976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.962977 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:26.963020 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.963021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.963973 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.963974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.965356 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.965359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.966179 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.966180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.966277 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.966292 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.966385 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.966451 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":86,"thread_start_us":70,"threads_started":1}
I20260925 14:54:26.967401 19670 tablet_service.cc:1467] Processing CreateTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 (DEFAULT_TABLE table=test-table [id=cee2ddfc538f492d860001a8daa19d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:26.967556 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37611 (local address 127.0.0.1:36106)
0925 14:54:26.962331 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.962359 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.962401 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.962724 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.962727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.962730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:26.962799 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.962802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.964234 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.964235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.965374 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.965376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.967305 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.967307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.967369 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.967373 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.967446 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.967485 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":52,"spinlock_wait_cycles":768,"thread_start_us":81,"threads_started":1}
I20260925 14:54:26.967568 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.962144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.0.98.2:37611)
0925 14:54:26.962539 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.962544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.962553 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:26.962595 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.962597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.962613 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.962692 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:26.962977 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.964126 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.966269 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.967041 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.967285 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.967349 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.967461 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.967500 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":86,"threads_started":1}
I20260925 14:54:26.967607 19938 tablet_service.cc:1467] Processing CreateTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 (DEFAULT_TABLE table=test-table [id=cee2ddfc538f492d860001a8daa19d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:26.967778 19670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c49dd4f9f94bb7bc72408324412dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:26.967877 19938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c49dd4f9f94bb7bc72408324412dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:26.970178 20022 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: Bootstrap starting.
I20260925 14:54:26.970883 20022 tablet_bootstrap.cc:654] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:26.970963 19804 tablet_service.cc:1467] Processing CreateTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 (DEFAULT_TABLE table=test-table [id=cee2ddfc538f492d860001a8daa19d92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:26.971266 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8c49dd4f9f94bb7bc72408324412dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:26.971434 20022 log.cc:824] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:26.971783 20021 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Bootstrap starting.
I20260925 14:54:26.972234 20022 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: No bootstrap required, opened a new log
I20260925 14:54:26.972321 20022 ts_tablet_manager.cc:1397] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:26.972390 20021 tablet_bootstrap.cc:654] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:26.972853 20021 log.cc:824] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:26.973568 20021 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: No bootstrap required, opened a new log
I20260925 14:54:26.973640 20021 ts_tablet_manager.cc:1397] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:26.974511 20025 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Bootstrap starting.
I20260925 14:54:26.974463 20022 raft_consensus.cc:348] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.974694 20022 raft_consensus.cc:374] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:26.974746 20022 raft_consensus.cc:729] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3560ec5f134c7194b58f028fdf11f3, State: Initialized, Role: FOLLOWER
I20260925 14:54:26.974859 20022 consensus_queue.cc:260] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.975149 20022 ts_tablet_manager.cc:1428] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:26.975235 20025 tablet_bootstrap.cc:654] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:26.975281 19736 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260925 14:54:26.975467 20021 raft_consensus.cc:348] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.975652 20021 raft_consensus.cc:374] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:26.975693 20021 raft_consensus.cc:729] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66299b117ea41a880c16e53ba7ed083, State: Initialized, Role: FOLLOWER
I20260925 14:54:26.975805 20021 consensus_queue.cc:260] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.976074 20021 ts_tablet_manager.cc:1428] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:54:26.976126 20025 log.cc:824] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:26.976227 20004 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260925 14:54:26.977183 20025 tablet_bootstrap.cc:492] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: No bootstrap required, opened a new log
I20260925 14:54:26.977234 20025 ts_tablet_manager.cc:1397] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:54:26.978574 20025 raft_consensus.cc:348] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.978694 20025 raft_consensus.cc:374] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:26.978715 20025 raft_consensus.cc:729] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab987c2bc1d4b3fa2c55273d584b2fa, State: Initialized, Role: FOLLOWER
I20260925 14:54:26.978818 20025 consensus_queue.cc:260] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.979038 20025 ts_tablet_manager.cc:1428] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:54:26.979099 19870 heartbeater.cc:503] Master 127.0.98.62:44675 was elected leader, sending a full tablet report...
I20260925 14:54:26.989100 20027 raft_consensus.cc:482] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:26.989192 20027 raft_consensus.cc:504] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.989485 20027 leader_election.cc:288] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3560ec5f134c7194b58f028fdf11f3 (127.0.98.1:41911), cab987c2bc1d4b3fa2c55273d584b2fa (127.0.98.2:37611)
I20260925 14:54:26.993254 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.989676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41911 (local address 127.0.98.3:54827)
0925 14:54:26.989918 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.989945 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.990000 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.990126 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.990127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.990128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:26.990187 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.990188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.991035 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.991035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.992207 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.992210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.992974 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.992975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.993066 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.993070 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.993113 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.993147 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":82,"thread_start_us":117,"threads_started":1}
I20260925 14:54:26.993294 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.989781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54827 (local address 127.0.98.1:41911)
0925 14:54:26.989828 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.989830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.990014 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:54:26.990048 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.990049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.990059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.990127 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:26.990259 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.990959 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.992327 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.992893 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.992964 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.992974 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.993132 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.993182 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:26.993340 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.989798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34369 (local address 127.0.98.2:37611)
0925 14:54:26.989845 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:26.989848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:26.989856 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:26.989882 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:26.989883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:26.989894 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:26.989954 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:26.990118 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.990896 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.992327 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.992898 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.992964 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.992974 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:26.993218 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:54:26.993270 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:54:26.993373 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:26.989676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37611 (local address 127.0.98.3:34369)
0925 14:54:26.989740 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:26.989755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:26.989814 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:26.989970 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:26.989971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:26.989973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:26.990050 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.990052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.990963 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.990964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.992207 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:26.992210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:26.993011 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:26.993012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:26.993142 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:26.993145 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:26.993200 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:54:26.993240 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260925 14:54:26.993836 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c49dd4f9f94bb7bc72408324412dd2" candidate_uuid: "c66299b117ea41a880c16e53ba7ed083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3560ec5f134c7194b58f028fdf11f3" is_pre_election: true
I20260925 14:54:26.993865 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c49dd4f9f94bb7bc72408324412dd2" candidate_uuid: "c66299b117ea41a880c16e53ba7ed083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" is_pre_election: true
I20260925 14:54:26.993988 19690 raft_consensus.cc:2454] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c66299b117ea41a880c16e53ba7ed083 in term 0.
I20260925 14:54:26.994000 19824 raft_consensus.cc:2454] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c66299b117ea41a880c16e53ba7ed083 in term 0.
I20260925 14:54:26.994228 19894 leader_election.cc:302] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c66299b117ea41a880c16e53ba7ed083, cab987c2bc1d4b3fa2c55273d584b2fa; no voters:
I20260925 14:54:26.994400 20027 raft_consensus.cc:2781] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:26.994472 20027 raft_consensus.cc:482] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:26.994506 20027 raft_consensus.cc:3037] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:26.995194 20027 raft_consensus.cc:504] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.995366 20027 leader_election.cc:288] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3560ec5f134c7194b58f028fdf11f3 (127.0.98.1:41911), cab987c2bc1d4b3fa2c55273d584b2fa (127.0.98.2:37611)
I20260925 14:54:26.995505 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c49dd4f9f94bb7bc72408324412dd2" candidate_uuid: "c66299b117ea41a880c16e53ba7ed083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa"
I20260925 14:54:26.995527 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c49dd4f9f94bb7bc72408324412dd2" candidate_uuid: "c66299b117ea41a880c16e53ba7ed083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3560ec5f134c7194b58f028fdf11f3"
I20260925 14:54:26.995606 19824 raft_consensus.cc:3037] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:26.995602 19690 raft_consensus.cc:3037] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:26.996304 19690 raft_consensus.cc:2454] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c66299b117ea41a880c16e53ba7ed083 in term 1.
I20260925 14:54:26.996369 19824 raft_consensus.cc:2454] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c66299b117ea41a880c16e53ba7ed083 in term 1.
I20260925 14:54:26.996508 19893 leader_election.cc:302] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3560ec5f134c7194b58f028fdf11f3, c66299b117ea41a880c16e53ba7ed083; no voters:
I20260925 14:54:26.996621 20027 raft_consensus.cc:2781] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:26.996783 20027 raft_consensus.cc:686] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 1 LEADER]: Becoming Leader. State: Replica: c66299b117ea41a880c16e53ba7ed083, State: Running, Role: LEADER
I20260925 14:54:26.996887 20027 consensus_queue.cc:237] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } }
I20260925 14:54:26.997722 19545 catalog_manager.cc:5495] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 reported cstate change: term changed from 0 to 1, leader changed from <none> to c66299b117ea41a880c16e53ba7ed083 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c66299b117ea41a880c16e53ba7ed083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c66299b117ea41a880c16e53ba7ed083" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } health_report { overall_health: HEALTHY } } }
I20260925 14:54:27.004317 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.002135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.0.98.3:41345)
0925 14:54:27.002184 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.002185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.002194 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:27.002226 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.002229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.002249 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.002306 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:27.002503 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.002968 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.003710 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.003885 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.003946 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.004053 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:27.004148 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:27.004201 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.004230 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.004245 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:27.005985 19690 raft_consensus.cc:1261] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 1 FOLLOWER]: Refusing update from remote peer c66299b117ea41a880c16e53ba7ed083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:27.005986 19824 raft_consensus.cc:1261] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 1 FOLLOWER]: Refusing update from remote peer c66299b117ea41a880c16e53ba7ed083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:27.006332 20027 consensus_queue.cc:1035] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3560ec5f134c7194b58f028fdf11f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:27.006498 20031 consensus_queue.cc:1035] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab987c2bc1d4b3fa2c55273d584b2fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:27.008975 20034 mvcc.cc:205] Tried to move back new op lower bound from 7333265682454106112 to 7333265682420097024. Current Snapshot: MvccSnapshot[applied={T|T < 7333265682454106112}]
I20260925 14:54:27.009471 20036 mvcc.cc:205] Tried to move back new op lower bound from 7333265682454106112 to 7333265682420097024. Current Snapshot: MvccSnapshot[applied={T|T < 7333265682454106112}]
I20260925 14:54:27.009917 20035 mvcc.cc:205] Tried to move back new op lower bound from 7333265682454106112 to 7333265682420097024. Current Snapshot: MvccSnapshot[applied={T|T < 7333265682454106112}]
I20260925 14:54:27.011825 19545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:27.011914 19545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:27.012646 19545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49d83ca6a976422db53d4e243965bed4: Sending DeleteTablet for 3 replicas of tablet a8c49dd4f9f94bb7bc72408324412dd2
I20260925 14:54:27.012969 19670 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35526
I20260925 14:54:27.012976 19938 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49572
I20260925 14:54:27.013062 19804 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c49dd4f9f94bb7bc72408324412dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36106
I20260925 14:54:27.013248 20046 tablet_replica.cc:330] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: stopping tablet replica
I20260925 14:54:27.013324 20044 tablet_replica.cc:330] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3: stopping tablet replica
I20260925 14:54:27.013347 20046 raft_consensus.cc:2229] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:27.013461 20046 raft_consensus.cc:2258] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.013479 20044 raft_consensus.cc:2229] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:27.013577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19607
I20260925 14:54:27.013707 20044 raft_consensus.cc:2258] T a8c49dd4f9f94bb7bc72408324412dd2 P 1b3560ec5f134c7194b58f028fdf11f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.014401 20046 ts_tablet_manager.cc:1905] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:27.015859 20045 tablet_replica.cc:330] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: stopping tablet replica
I20260925 14:54:27.015998 20045 raft_consensus.cc:2229] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:27.016151 20045 raft_consensus.cc:2258] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.016314 20046 ts_tablet_manager.cc:1918] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:27.016412 20046 log.cc:1196] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/a8c49dd4f9f94bb7bc72408324412dd2
I20260925 14:54:27.016674 20046 ts_tablet_manager.cc:1939] T a8c49dd4f9f94bb7bc72408324412dd2 P c66299b117ea41a880c16e53ba7ed083: Deleting consensus metadata
I20260925 14:54:27.016893 20045 ts_tablet_manager.cc:1905] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:27.017082 19531 catalog_manager.cc:4841] TS c66299b117ea41a880c16e53ba7ed083 (127.0.98.3:41345): tablet a8c49dd4f9f94bb7bc72408324412dd2 (table test-table [id=cee2ddfc538f492d860001a8daa19d92]) successfully deleted
I20260925 14:54:27.018630 20045 ts_tablet_manager.cc:1918] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:27.018755 20045 log.cc:1196] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/a8c49dd4f9f94bb7bc72408324412dd2
I20260925 14:54:27.019080 20045 ts_tablet_manager.cc:1939] T a8c49dd4f9f94bb7bc72408324412dd2 P cab987c2bc1d4b3fa2c55273d584b2fa: Deleting consensus metadata
I20260925 14:54:27.019543 19532 catalog_manager.cc:4841] TS cab987c2bc1d4b3fa2c55273d584b2fa (127.0.98.2:37611): tablet a8c49dd4f9f94bb7bc72408324412dd2 (table test-table [id=cee2ddfc538f492d860001a8daa19d92]) successfully deleted
W20260925 14:54:27.019891 19531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41911 (error 108)
I20260925 14:54:27.020141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19741
I20260925 14:54:27.020712 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.020260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41911 (local address 127.0.0.1:35540)
0925 14:54:27.020637 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.020681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260925 14:54:27.021589 19531 catalog_manager.cc:4584] TS 1b3560ec5f134c7194b58f028fdf11f3 (127.0.98.1:41911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8c49dd4f9f94bb7bc72408324412dd2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41911: connect: Connection refused (error 111)
I20260925 14:54:27.024538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19875
I20260925 14:54:27.029403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 19513
2026-09-25T14:54:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1911 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T14:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:27Z Disabled control of system clock
I20260925 14:54:27.056795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46337
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:27.136269 20051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.136490 20051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.136511 20051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.137692 20051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:27.137753 20051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.137769 20051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.137782 20051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:27.137796 20051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:27.139180 20051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:27.139371 20051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.139617 20051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.144409 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.144387 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.144387 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.144586 20051 server_base.cc:1034] running on GCE node
I20260925 14:54:27.144886 20051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.145190 20051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.146368 20051 hybrid_clock.cc:648] HybridClock initialized: now 1790348067146314 us; error 41 us; skew 500 ppm
I20260925 14:54:27.147701 20051 webserver.cc:460] Webserver started at http://127.0.98.62:36291/ using document root <none> and password file <none>
I20260925 14:54:27.147970 20051 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.148046 20051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.148195 20051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.149281 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "df12a748c8cd4c0b9075c24482732288"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "4667545137087c58872a8aa516576b8d"
server_key_iv: "9cc4eddc474e6bf3c0783720df18eb80"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.149641 20051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "df12a748c8cd4c0b9075c24482732288"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "4667545137087c58872a8aa516576b8d"
server_key_iv: "9cc4eddc474e6bf3c0783720df18eb80"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.150998 20051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:27.151758 20065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.151935 20051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:27.152035 20051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "df12a748c8cd4c0b9075c24482732288"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "4667545137087c58872a8aa516576b8d"
server_key_iv: "9cc4eddc474e6bf3c0783720df18eb80"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.152112 20051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.161772 20051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.162055 20051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.162207 20051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.166119 20051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46337
I20260925 14:54:27.166148 20117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46337 every 8 connection(s)
I20260925 14:54:27.166550 20051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:27.167512 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:27.169663 20118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: Bootstrap starting.
I20260925 14:54:27.170184 20118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:27.170507 20118 log.cc:824] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:27.170796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20051
I20260925 14:54:27.170930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:27.171123 392 external_mini_cluster.cc:1428] Setting key 6c4d7e7b1d225672ad00a08f3c7d41a7
I20260925 14:54:27.171288 20118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: No bootstrap required, opened a new log
I20260925 14:54:27.173774 20118 raft_consensus.cc:348] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260925 14:54:27.173941 20118 raft_consensus.cc:374] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:27.173990 20118 raft_consensus.cc:729] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df12a748c8cd4c0b9075c24482732288, State: Initialized, Role: FOLLOWER
I20260925 14:54:27.174121 20118 consensus_queue.cc:260] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260925 14:54:27.174206 20118 raft_consensus.cc:388] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:27.174235 20118 raft_consensus.cc:482] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:27.174274 20118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:27.174948 20118 raft_consensus.cc:504] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260925 14:54:27.175067 20118 leader_election.cc:302] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df12a748c8cd4c0b9075c24482732288; no voters:
I20260925 14:54:27.175236 20118 leader_election.cc:288] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:27.175304 20123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:27.175467 20123 raft_consensus.cc:686] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [term 1 LEADER]: Becoming Leader. State: Replica: df12a748c8cd4c0b9075c24482732288, State: Running, Role: LEADER
I20260925 14:54:27.175487 20118 sys_catalog.cc:564] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:27.175555 20123 consensus_queue.cc:237] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } }
I20260925 14:54:27.175971 20124 sys_catalog.cc:455] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df12a748c8cd4c0b9075c24482732288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } } }
I20260925 14:54:27.176059 20124 sys_catalog.cc:458] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:27.176498 20125 sys_catalog.cc:455] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df12a748c8cd4c0b9075c24482732288. Latest consensus state: current_term: 1 leader_uuid: "df12a748c8cd4c0b9075c24482732288" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df12a748c8cd4c0b9075c24482732288" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46337 } } }
I20260925 14:54:27.176559 20132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:27.176581 20125 sys_catalog.cc:458] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:27.177057 20132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:27.177795 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.171666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.98.62:46337)
0925 14:54:27.171873 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.171881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.171907 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:54:27.172557 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.172563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.172622 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.172729 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.173002 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.173937 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.175151 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.175776 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.175897 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.176834 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.176838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.176839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.177053 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.177097 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.177112 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.177185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.177490 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.177571 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260925 14:54:27.178820 20132 catalog_manager.cc:1347] Generated new cluster ID: af8370db2a394026a749481f0ff3b449
I20260925 14:54:27.178877 20132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:27.190068 20132 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:27.190667 20132 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:27.201606 20132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: Generated new TSK 0
I20260925 14:54:27.201848 20132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:27.205049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:27.285312 20142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.285518 20142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.285539 20142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.286720 20142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.286760 20142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.286785 20142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:54:27.288153 20142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:27.288405 20142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.288697 20142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.294661 20147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.294677 20148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.294826 20150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.295048 20142 server_base.cc:1034] running on GCE node
I20260925 14:54:27.295235 20142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.295475 20142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.296643 20142 hybrid_clock.cc:648] HybridClock initialized: now 1790348067296620 us; error 33 us; skew 500 ppm
I20260925 14:54:27.298022 20142 webserver.cc:460] Webserver started at http://127.0.98.1:45477/ using document root <none> and password file <none>
I20260925 14:54:27.298269 20142 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.298326 20142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.298472 20142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.299944 20142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data/instance:
uuid: "1641ea408d2142d0b672e3cea17e415e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "afaba45f2efbb2768a2ad43abf97c7ad"
server_key_iv: "abefe04a87a6c9ee867bc49c7f6b7335"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.300375 20142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1641ea408d2142d0b672e3cea17e415e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "afaba45f2efbb2768a2ad43abf97c7ad"
server_key_iv: "abefe04a87a6c9ee867bc49c7f6b7335"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.301896 20142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:27.302735 20156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.302930 20142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:27.303028 20142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal
uuid: "1641ea408d2142d0b672e3cea17e415e"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "afaba45f2efbb2768a2ad43abf97c7ad"
server_key_iv: "abefe04a87a6c9ee867bc49c7f6b7335"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.303097 20142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.312008 20142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.312239 20142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.312341 20142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.312505 20142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.312747 20142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.312775 20142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.312795 20142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.312808 20142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.318564 20142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43731
I20260925 14:54:27.318619 20269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43731 every 8 connection(s)
I20260925 14:54:27.318936 20142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:54:27.319994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20142
I20260925 14:54:27.320107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:54:27.320299 392 external_mini_cluster.cc:1428] Setting key 85818e7504d1985ca000fe1095bded87
I20260925 14:54:27.321184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:54:27.325088 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.319639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33261 (local address 127.0.98.62:46337)
0925 14:54:27.319712 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.319715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.319958 (+ 243us) server_negotiation.cc:407] Connection header received
0925 14:54:27.320637 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.320639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.320656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.320708 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.321492 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.321995 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.323454 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.323776 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.323854 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.324749 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.324751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.324751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.324756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.324774 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.324779 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.324859 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.324930 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.324971 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:54:27.325165 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.319633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.1:33261)
0925 14:54:27.319841 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.319874 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.320529 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.320773 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.320778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.320789 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.321405 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.321410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.322091 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.322092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.323365 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.323367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.323885 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.323886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.324010 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.324581 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.324582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.324623 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.324650 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.324858 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.324860 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.324862 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.324908 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.324994 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260925 14:54:27.325676 20270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260925 14:54:27.325774 20270 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.325959 20270 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260925 14:54:27.326524 20082 ts_manager.cc:194] Registered new tserver with Master: 1641ea408d2142d0b672e3cea17e415e (127.0.98.1:43731)
I20260925 14:54:27.327553 20082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33261
W20260925 14:54:27.401345 20273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.401541 20273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.401562 20273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.402655 20273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.402688 20273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.402712 20273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:54:27.403967 20273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:27.404228 20273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.404541 20273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.409437 20279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.409451 20278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.409435 20281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.409853 20273 server_base.cc:1034] running on GCE node
I20260925 14:54:27.410043 20273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.410233 20273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.411366 20273 hybrid_clock.cc:648] HybridClock initialized: now 1790348067411360 us; error 36 us; skew 500 ppm
I20260925 14:54:27.412572 20273 webserver.cc:460] Webserver started at http://127.0.98.2:34353/ using document root <none> and password file <none>
I20260925 14:54:27.412811 20273 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.412880 20273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.413009 20273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.414016 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data/instance:
uuid: "ed26c1103a1040789eab6cb391728842"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "14d5f1ce04dff90da5c4a8f04b020667"
server_key_iv: "7921a02655c39de91cf0616f8d62e399"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.414317 20273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ed26c1103a1040789eab6cb391728842"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "14d5f1ce04dff90da5c4a8f04b020667"
server_key_iv: "7921a02655c39de91cf0616f8d62e399"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.415591 20273 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:27.416324 20287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.416508 20273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:27.416590 20273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal
uuid: "ed26c1103a1040789eab6cb391728842"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "14d5f1ce04dff90da5c4a8f04b020667"
server_key_iv: "7921a02655c39de91cf0616f8d62e399"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.416676 20273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.429147 20273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.429450 20273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.429608 20273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.429849 20273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.430181 20273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.430222 20273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.430267 20273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.430297 20273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.436432 20273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41111
I20260925 14:54:27.436522 20400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41111 every 8 connection(s)
I20260925 14:54:27.436843 20273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:54:27.442154 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.437582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43901 (local address 127.0.98.62:46337)
0925 14:54:27.437653 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.437655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.437912 (+ 257us) server_negotiation.cc:407] Connection header received
0925 14:54:27.438532 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.438533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.438546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.438619 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.439298 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.439786 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.440805 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.440983 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.441036 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.441936 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.441937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.441937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.441941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.441951 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.441954 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.441994 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.442075 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.442096 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:54:27.442366 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.437572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.2:43901)
0925 14:54:27.437759 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.437804 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.438408 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.438676 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.438680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.438687 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.439184 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.439189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.439893 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.439893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.440720 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.440722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.441076 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.441077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.441192 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.441813 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.441813 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.441843 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.441865 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.442002 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.442004 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.442007 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.442052 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.442132 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260925 14:54:27.442909 20401 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260925 14:54:27.443010 20401 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.443189 20401 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260925 14:54:27.443578 20082 ts_manager.cc:194] Registered new tserver with Master: ed26c1103a1040789eab6cb391728842 (127.0.98.2:41111)
I20260925 14:54:27.443958 20082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43901
I20260925 14:54:27.446521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20273
I20260925 14:54:27.446602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:54:27.446750 392 external_mini_cluster.cc:1428] Setting key 3effdbe42ef5d3278fee82da61282c4d
I20260925 14:54:27.447588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--builtin_ntp_servers=127.0.98.20:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:54:27.527268 20404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:27.527478 20404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:27.527499 20404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:27.528568 20404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:27.528601 20404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:27.528626 20404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:54:27.529860 20404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:27.530047 20404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:27.530285 20404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:27.535454 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.535413 20404 server_base.cc:1034] running on GCE node
W20260925 14:54:27.535393 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:27.535393 20412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:27.535880 20404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:27.536156 20404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:27.537318 20404 hybrid_clock.cc:648] HybridClock initialized: now 1790348067537308 us; error 43 us; skew 500 ppm
I20260925 14:54:27.538733 20404 webserver.cc:460] Webserver started at http://127.0.98.3:39367/ using document root <none> and password file <none>
I20260925 14:54:27.538983 20404 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:27.539052 20404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:27.539201 20404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:27.540251 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0baf2f0d60b43c0b05c1af80dea376b"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "3aa0d6960aa9acd748f5aea898a5b99d"
server_key_iv: "700c2b6513dc0a29e0bebbc0d17753c9"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.540616 20404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0baf2f0d60b43c0b05c1af80dea376b"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "3aa0d6960aa9acd748f5aea898a5b99d"
server_key_iv: "700c2b6513dc0a29e0bebbc0d17753c9"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.541966 20404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:27.542797 20418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.542975 20404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:27.543068 20404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal
uuid: "e0baf2f0d60b43c0b05c1af80dea376b"
format_stamp: "Formatted at 2026-09-25 14:54:27 on dist-test-slave-gpc1"
server_key: "3aa0d6960aa9acd748f5aea898a5b99d"
server_key_iv: "700c2b6513dc0a29e0bebbc0d17753c9"
server_key_version: "encryptionkey@0"
I20260925 14:54:27.543151 20404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:27.563081 20404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:27.563397 20404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:27.563552 20404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:27.563786 20404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:54:27.564121 20404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:54:27.564172 20404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.564206 20404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:54:27.564232 20404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:27.570153 20404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37863
I20260925 14:54:27.570209 20531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37863 every 8 connection(s)
I20260925 14:54:27.570581 20404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:54:27.572319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20404
I20260925 14:54:27.572408 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:54:27.572621 392 external_mini_cluster.cc:1428] Setting key 108afcbc208386fd62df8482b28f93b7
I20260925 14:54:27.575975 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.571342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52419 (local address 127.0.98.62:46337)
0925 14:54:27.571414 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.571416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.571659 (+ 243us) server_negotiation.cc:407] Connection header received
0925 14:54:27.572146 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.572147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.572160 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.572245 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.573120 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.573759 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.574700 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.574874 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.574930 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.575676 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.575678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.575679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.575683 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.575698 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.575702 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.575763 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.575891 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.575911 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:54:27.576232 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46337 (local address 127.0.98.3:52419)
0925 14:54:27.571538 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.571572 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.572053 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.572301 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.572306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.572314 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:27.573027 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.573035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.573844 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.573845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.574634 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.574636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.574944 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.574945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.575046 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.575539 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:54:27.575539 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:27.575567 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:54:27.575588 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:27.575789 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:27.575800 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:27.575804 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.575871 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.575988 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260925 14:54:27.576700 20532 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46337
I20260925 14:54:27.576812 20532 heartbeater.cc:464] Registering TS with master...
I20260925 14:54:27.577008 20532 heartbeater.cc:511] Master 127.0.98.62:46337 requested a full tablet report, sending...
I20260925 14:54:27.577359 20082 ts_manager.cc:194] Registered new tserver with Master: e0baf2f0d60b43c0b05c1af80dea376b (127.0.98.3:37863)
I20260925 14:54:27.577682 20082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52419
I20260925 14:54:27.584290 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:54:27.588382 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.585670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.0.98.62:46337)
0925 14:54:27.585733 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.585735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.585832 (+ 97us) server_negotiation.cc:407] Connection header received
0925 14:54:27.585884 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.585885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.585893 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.585927 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:27.586069 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.586603 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.587346 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.587520 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.587572 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.588225 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:27.588226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:27.588227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:27.588230 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:27.588240 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:27.588243 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:27.588280 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.588311 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.588327 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:54:27.590032 20082 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:54:27.591645 20082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:54:27.597935 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.593827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43731 (local address 127.0.0.1:41038)
0925 14:54:27.594069 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.594079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.594131 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.594519 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.594520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.594522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:27.594552 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.594553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595787 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.595788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.596788 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.596790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.597685 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.597686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.597740 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.597751 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.597791 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.597830 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":42,"thread_start_us":100,"threads_started":1}
I20260925 14:54:27.597951 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.594131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41038 (local address 127.0.98.1:43731)
0925 14:54:27.594285 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.594290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.594299 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:27.594337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.594359 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.594375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.594448 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:27.594629 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595704 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.596874 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.597600 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.597680 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.597774 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.597819 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.597866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260925 14:54:27.598110 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.594304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.0.98.3:37863)
0925 14:54:27.594651 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.594655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.594664 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:27.594703 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.594705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.594723 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.594812 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:27.594911 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595577 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.597110 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.597798 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.597863 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.597958 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.597988 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.598031 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":90,"threads_started":1}
I20260925 14:54:27.598508 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.593991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37863 (local address 127.0.0.1:59016)
0925 14:54:27.594482 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.594492 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.594536 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.594810 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.594811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.594813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:27.594855 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.594856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595661 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.595662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.597023 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.597026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.597877 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.597878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.597931 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.597934 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.597978 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.598442 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":93,"thread_start_us":70,"threads_started":1}
I20260925 14:54:27.598848 20204 tablet_service.cc:1467] Processing CreateTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 (DEFAULT_TABLE table=test-table [id=56ed72ee435e415397734cbcdb471ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:27.599227 20204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bb89c86b1e4f7b979d0d5fb6075d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:27.599151 20466 tablet_service.cc:1467] Processing CreateTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 (DEFAULT_TABLE table=test-table [id=56ed72ee435e415397734cbcdb471ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:27.599449 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.593783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41111 (local address 127.0.0.1:57078)
0925 14:54:27.593980 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.594028 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.594094 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.594372 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.594375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.594378 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:27.594456 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.594458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595728 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.595729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.597023 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.597026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.599220 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.599221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.599287 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.599292 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.599342 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.599380 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":99,"thread_start_us":68,"threads_started":1}
I20260925 14:54:27.599465 20466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bb89c86b1e4f7b979d0d5fb6075d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:27.599758 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.593888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.98.2:41111)
0925 14:54:27.594055 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.594060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.594072 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:54:27.594176 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.594179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.594194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.594267 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:27.594538 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.595655 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.598251 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.599050 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.599115 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.599191 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.599640 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.599685 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260925 14:54:27.600344 20335 tablet_service.cc:1467] Processing CreateTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 (DEFAULT_TABLE table=test-table [id=56ed72ee435e415397734cbcdb471ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:54:27.600666 20335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bb89c86b1e4f7b979d0d5fb6075d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:27.602674 20547 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Bootstrap starting.
I20260925 14:54:27.603402 20547 tablet_bootstrap.cc:654] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:27.603482 20548 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Bootstrap starting.
I20260925 14:54:27.603754 20547 log.cc:824] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:27.604096 20548 tablet_bootstrap.cc:654] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:27.604140 20546 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: Bootstrap starting.
I20260925 14:54:27.604391 20548 log.cc:824] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:27.604547 20547 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: No bootstrap required, opened a new log
I20260925 14:54:27.604615 20547 ts_tablet_manager.cc:1397] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:27.604782 20546 tablet_bootstrap.cc:654] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:27.605116 20548 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: No bootstrap required, opened a new log
I20260925 14:54:27.605100 20546 log.cc:824] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:27.605195 20548 ts_tablet_manager.cc:1397] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:27.605893 20546 tablet_bootstrap.cc:492] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: No bootstrap required, opened a new log
I20260925 14:54:27.605967 20546 ts_tablet_manager.cc:1397] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:54:27.607448 20547 raft_consensus.cc:348] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.607614 20547 raft_consensus.cc:374] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:27.607513 20548 raft_consensus.cc:348] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.607651 20547 raft_consensus.cc:729] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0baf2f0d60b43c0b05c1af80dea376b, State: Initialized, Role: FOLLOWER
I20260925 14:54:27.607661 20548 raft_consensus.cc:374] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:27.607694 20548 raft_consensus.cc:729] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed26c1103a1040789eab6cb391728842, State: Initialized, Role: FOLLOWER
I20260925 14:54:27.607790 20547 consensus_queue.cc:260] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.607822 20548 consensus_queue.cc:260] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.608047 20547 ts_tablet_manager.cc:1428] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:54:27.608088 20548 ts_tablet_manager.cc:1428] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:27.608175 20532 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
I20260925 14:54:27.608112 20401 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
I20260925 14:54:27.608597 20546 raft_consensus.cc:348] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.608743 20546 raft_consensus.cc:374] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:27.608783 20546 raft_consensus.cc:729] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1641ea408d2142d0b672e3cea17e415e, State: Initialized, Role: FOLLOWER
I20260925 14:54:27.608888 20546 consensus_queue.cc:260] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.609143 20546 ts_tablet_manager.cc:1428] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:54:27.609194 20270 heartbeater.cc:503] Master 127.0.98.62:46337 was elected leader, sending a full tablet report...
W20260925 14:54:27.687436 20402 tablet.cc:2369] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:54:27.714211 20552 raft_consensus.cc:482] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:54:27.714378 20552 raft_consensus.cc:504] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.714699 20552 leader_election.cc:288] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed26c1103a1040789eab6cb391728842 (127.0.98.2:41111), 1641ea408d2142d0b672e3cea17e415e (127.0.98.1:43731)
I20260925 14:54:27.718605 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.714965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33909 (local address 127.0.98.1:43731)
0925 14:54:27.715011 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.715013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.715053 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:54:27.715082 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.715083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.715095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.715152 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:27.715358 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.716145 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.717653 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.718216 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.718294 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.718305 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.718492 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.718534 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:54:27.718605 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.715032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54711 (local address 127.0.98.2:41111)
0925 14:54:27.715071 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.715074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.715188 (+ 114us) server_negotiation.cc:407] Connection header received
0925 14:54:27.715234 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.715235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.715244 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.715283 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:54:27.715450 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.716143 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.717653 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.718216 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.718300 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.718308 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.718493 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.718534 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:27.718624 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.714882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43731 (local address 127.0.98.3:33909)
0925 14:54:27.714947 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.714958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.715002 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.715172 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.715173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.715175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:27.715252 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.715254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.716227 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.716228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.717554 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.717558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.718304 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.718305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.718410 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.718414 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.718474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.718525 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":61}
I20260925 14:54:27.718636 20555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.714893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41111 (local address 127.0.98.3:54711)
0925 14:54:27.715107 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.715116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:27.715173 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:27.715310 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:27.715311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:27.715313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:54:27.715375 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.715377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.716227 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.716228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.717554 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:27.717558 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.718304 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.718305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:27.718410 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.718424 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:54:27.718474 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:27.718525 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":140,"thread_start_us":100,"threads_started":1}
I20260925 14:54:27.719061 20224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bb89c86b1e4f7b979d0d5fb6075d50" candidate_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1641ea408d2142d0b672e3cea17e415e" is_pre_election: true
I20260925 14:54:27.719141 20355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bb89c86b1e4f7b979d0d5fb6075d50" candidate_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed26c1103a1040789eab6cb391728842" is_pre_election: true
I20260925 14:54:27.719244 20224 raft_consensus.cc:2454] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0baf2f0d60b43c0b05c1af80dea376b in term 0.
I20260925 14:54:27.719257 20355 raft_consensus.cc:2454] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0baf2f0d60b43c0b05c1af80dea376b in term 0.
I20260925 14:54:27.719480 20419 leader_election.cc:302] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0baf2f0d60b43c0b05c1af80dea376b, ed26c1103a1040789eab6cb391728842; no voters:
I20260925 14:54:27.719601 20552 raft_consensus.cc:2781] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:54:27.719671 20552 raft_consensus.cc:482] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:54:27.719693 20552 raft_consensus.cc:3037] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:27.720479 20552 raft_consensus.cc:504] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.720623 20552 leader_election.cc:288] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [CANDIDATE]: Term 1 election: Requested vote from peers ed26c1103a1040789eab6cb391728842 (127.0.98.2:41111), 1641ea408d2142d0b672e3cea17e415e (127.0.98.1:43731)
I20260925 14:54:27.720777 20355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bb89c86b1e4f7b979d0d5fb6075d50" candidate_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed26c1103a1040789eab6cb391728842"
I20260925 14:54:27.720781 20224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bb89c86b1e4f7b979d0d5fb6075d50" candidate_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1641ea408d2142d0b672e3cea17e415e"
I20260925 14:54:27.720860 20224 raft_consensus.cc:3037] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:27.720876 20355 raft_consensus.cc:3037] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:27.721588 20355 raft_consensus.cc:2454] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0baf2f0d60b43c0b05c1af80dea376b in term 1.
I20260925 14:54:27.721589 20224 raft_consensus.cc:2454] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0baf2f0d60b43c0b05c1af80dea376b in term 1.
I20260925 14:54:27.721750 20419 leader_election.cc:302] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0baf2f0d60b43c0b05c1af80dea376b, ed26c1103a1040789eab6cb391728842; no voters:
I20260925 14:54:27.721843 20552 raft_consensus.cc:2781] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:27.721990 20552 raft_consensus.cc:686] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 1 LEADER]: Becoming Leader. State: Replica: e0baf2f0d60b43c0b05c1af80dea376b, State: Running, Role: LEADER
I20260925 14:54:27.722086 20552 consensus_queue.cc:237] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } }
I20260925 14:54:27.722975 20081 catalog_manager.cc:5495] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0baf2f0d60b43c0b05c1af80dea376b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0baf2f0d60b43c0b05c1af80dea376b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 } health_report { overall_health: UNKNOWN } } }
I20260925 14:54:27.743880 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.741581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.0.98.3:37863)
0925 14:54:27.741645 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:27.741647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:27.741656 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:27.741719 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:27.741720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:27.741736 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:27.741788 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:27.741909 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.742455 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.743154 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:27.743357 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:27.743411 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:27.743544 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:27.743642 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:27.743701 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:27.743732 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:54:27.743750 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:27.745786 20355 raft_consensus.cc:1261] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 1 FOLLOWER]: Refusing update from remote peer e0baf2f0d60b43c0b05c1af80dea376b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:27.745786 20224 raft_consensus.cc:1261] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 1 FOLLOWER]: Refusing update from remote peer e0baf2f0d60b43c0b05c1af80dea376b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:27.746150 20552 consensus_queue.cc:1035] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed26c1103a1040789eab6cb391728842" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:27.746287 20556 consensus_queue.cc:1035] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1641ea408d2142d0b672e3cea17e415e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:27.750061 20561 mvcc.cc:205] Tried to move back new op lower bound from 7333265685483655168 to 7333265685390344192. Current Snapshot: MvccSnapshot[applied={T|T < 7333265685483655168 or (T in {7333265685483655168})}]
I20260925 14:54:27.750190 20560 mvcc.cc:205] Tried to move back new op lower bound from 7333265685483655168 to 7333265685390344192. Current Snapshot: MvccSnapshot[applied={T|T < 7333265685483655168 or (T in {7333265685483655168})}]
I20260925 14:54:27.751101 20081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:27.751179 20081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:54:27.751932 20081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df12a748c8cd4c0b9075c24482732288: Sending DeleteTablet for 3 replicas of tablet 25bb89c86b1e4f7b979d0d5fb6075d50
I20260925 14:54:27.752317 20466 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='slave'} at 127.0.0.1:59016
I20260925 14:54:27.752337 20335 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='slave'} at 127.0.0.1:57078
I20260925 14:54:27.752529 20569 tablet_replica.cc:330] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: stopping tablet replica
I20260925 14:54:27.752624 20569 raft_consensus.cc:2229] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:54:27.752628 20204 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bb89c86b1e4f7b979d0d5fb6075d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:54:27 UTC) from {username='slave'} at 127.0.0.1:41038
I20260925 14:54:27.752718 20569 raft_consensus.cc:2258] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.752964 20571 tablet_replica.cc:330] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e: stopping tablet replica
I20260925 14:54:27.753060 20571 raft_consensus.cc:2229] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:27.753058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20142
I20260925 14:54:27.753149 20571 raft_consensus.cc:2258] T 25bb89c86b1e4f7b979d0d5fb6075d50 P 1641ea408d2142d0b672e3cea17e415e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.753160 20569 ts_tablet_manager.cc:1905] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:27.755115 20569 ts_tablet_manager.cc:1918] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:27.755235 20569 log.cc:1196] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-2/wal/wals/25bb89c86b1e4f7b979d0d5fb6075d50
I20260925 14:54:27.755471 20569 ts_tablet_manager.cc:1939] T 25bb89c86b1e4f7b979d0d5fb6075d50 P e0baf2f0d60b43c0b05c1af80dea376b: Deleting consensus metadata
I20260925 14:54:27.755872 20066 catalog_manager.cc:4841] TS e0baf2f0d60b43c0b05c1af80dea376b (127.0.98.3:37863): tablet 25bb89c86b1e4f7b979d0d5fb6075d50 (table test-table [id=56ed72ee435e415397734cbcdb471ef5]) successfully deleted
I20260925 14:54:27.752893 20570 tablet_replica.cc:330] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: stopping tablet replica
I20260925 14:54:27.758641 20570 raft_consensus.cc:2229] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:54:27.758790 20570 raft_consensus.cc:2258] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:54:27.759196 20570 ts_tablet_manager.cc:1905] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:54:27.760669 20570 ts_tablet_manager.cc:1918] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:54:27.760752 20570 log.cc:1196] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Deleting WAL directory at /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347825060285-392-0/minicluster-data/ts-1/wal/wals/25bb89c86b1e4f7b979d0d5fb6075d50
W20260925 14:54:27.760800 20067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43731 (error 108)
I20260925 14:54:27.761013 20570 ts_tablet_manager.cc:1939] T 25bb89c86b1e4f7b979d0d5fb6075d50 P ed26c1103a1040789eab6cb391728842: Deleting consensus metadata
I20260925 14:54:27.761273 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:27.761148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43731 (local address 127.0.0.1:41050)
0925 14:54:27.761192 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:27.761238 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 14:54:27.761382 20067 catalog_manager.cc:4584] TS 1641ea408d2142d0b672e3cea17e415e (127.0.98.1:43731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25bb89c86b1e4f7b979d0d5fb6075d50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43731: connect: Connection refused (error 111)
I20260925 14:54:27.761478 20066 catalog_manager.cc:4841] TS ed26c1103a1040789eab6cb391728842 (127.0.98.2:41111): tablet 25bb89c86b1e4f7b979d0d5fb6075d50 (table test-table [id=56ed72ee435e415397734cbcdb471ef5]) successfully deleted
I20260925 14:54:27.761663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20273
I20260925 14:54:27.767073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20404
I20260925 14:54:27.772418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20051
2026-09-25T14:54:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (742 ms)
[----------] 6 tests from EncryptionPolicyTest (8479 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:27 dist-test-slave-gpc1 krb5kdc[20575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:27 dist-test-slave-gpc1 krb5kdc[20575](info): set up 2 sockets
Sep 25 14:54:27 dist-test-slave-gpc1 krb5kdc[20575](info): commencing operation
krb5kdc: starting...
W20260925 14:54:28.810323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:28 dist-test-slave-gpc1 krb5kdc[20575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:28.923412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:29.003149 20588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:29.003396 20588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:29.003433 20588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:29.004596 20588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:29.004667 20588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:29.004694 20588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:29.004726 20588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:29.004750 20588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:29.006239 20588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:29.006534 20588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:29.006755 20588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:29.011499 20593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:29.011482 20594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:29.011477 20596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:29.011886 20588 server_base.cc:1034] running on GCE node
I20260925 14:54:29.012151 20588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:29.012457 20588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:29.013595 20588 hybrid_clock.cc:648] HybridClock initialized: now 1790348069013585 us; error 30 us; skew 500 ppm
Sep 25 14:54:29 dist-test-slave-gpc1 krb5kdc[20575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:29.016080 20588 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:29.016525 20588 webserver.cc:460] Webserver started at http://127.0.0.1:44323/ using document root <none> and password file <none>
I20260925 14:54:29.016749 20588 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:29.016801 20588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:29.016955 20588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:29.018020 20588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "0abfc63870ca427a808f920f1c9b050a"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-gpc1"
server_key: "4d6526677b7d42a9d3a50241ae871a01"
server_key_iv: "e74d6b478162c93fa49bf41b4c047475"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.018448 20588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "0abfc63870ca427a808f920f1c9b050a"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-gpc1"
server_key: "4d6526677b7d42a9d3a50241ae871a01"
server_key_iv: "e74d6b478162c93fa49bf41b4c047475"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.019730 20588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:29.020439 20603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:29.020628 20588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:29.020721 20588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0abfc63870ca427a808f920f1c9b050a"
format_stamp: "Formatted at 2026-09-25 14:54:29 on dist-test-slave-gpc1"
server_key: "4d6526677b7d42a9d3a50241ae871a01"
server_key_iv: "e74d6b478162c93fa49bf41b4c047475"
server_key_version: "encryptionkey@0"
I20260925 14:54:29.020797 20588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:29.036118 20588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:29.037918 20588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:29.038067 20588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:29.042093 20588 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260925 14:54:29.042171 20655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260925 14:54:29.042582 20588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:29.043475 20656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:29.045723 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a: Bootstrap starting.
I20260925 14:54:29.046309 20656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:29.046662 20656 log.cc:824] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:29.047384 20656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a: No bootstrap required, opened a new log
I20260925 14:54:29.047832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20588
I20260925 14:54:29.047952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:29.048136 392 external_mini_cluster.cc:1428] Setting key 674f0c4d51576883f98f286b84ad302b
I20260925 14:54:29.049659 20656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 14:54:29.049832 20656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:29.049870 20656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abfc63870ca427a808f920f1c9b050a, State: Initialized, Role: FOLLOWER
I20260925 14:54:29.049993 20656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 14:54:29.050076 20656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:29.050097 20656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:29.050165 20656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:29.051301 20656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 14:54:29.051420 20656 leader_election.cc:302] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0abfc63870ca427a808f920f1c9b050a; no voters:
I20260925 14:54:29.051573 20656 leader_election.cc:288] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:29.051704 20661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:29.051805 20656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:29.051851 20661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [term 1 LEADER]: Becoming Leader. State: Replica: 0abfc63870ca427a808f920f1c9b050a, State: Running, Role: LEADER
I20260925 14:54:29.051937 20661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260925 14:54:29.052280 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0abfc63870ca427a808f920f1c9b050a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260925 14:54:29.052357 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:29.052556 20670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:29.052601 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0abfc63870ca427a808f920f1c9b050a. Latest consensus state: current_term: 1 leader_uuid: "0abfc63870ca427a808f920f1c9b050a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0abfc63870ca427a808f920f1c9b050a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260925 14:54:29.052752 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:29.053081 20670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:54:29 dist-test-slave-gpc1 krb5kdc[20575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:54:29.057745 20670 catalog_manager.cc:1347] Generated new cluster ID: 7f7c1435a309402fa7c1bfc3e77dcac4
I20260925 14:54:29.057803 20670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:29.060762 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:29.048728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55259 (local address 127.0.0.1:43105)
0925 14:54:29.048956 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:29.048966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:29.049001 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:54:29.049623 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:29.049632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:29.049697 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:29.049849 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:29.050682 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.052134 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.053371 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.053821 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.053981 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:29.056777 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:29.056782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:29.056782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:29.056795 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:29.058792 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.059208 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.059212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.059212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.059628 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.059875 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.059877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.059877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.060128 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:29.060309 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:29.060398 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:54:29.060608 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260925 14:54:29.061745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:29.065413 20670 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:29.066105 20670 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:29.073626 20670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0abfc63870ca427a808f920f1c9b050a: Generated new TSK 0
I20260925 14:54:29.073827 20670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:29.080042 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:29.063178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36547 (local address 127.0.0.1:43105)
0925 14:54:29.063224 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:29.063226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:29.065064 (+ 1838us) server_negotiation.cc:407] Connection header received
0925 14:54:29.065116 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:29.065117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:29.065127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:29.065170 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:29.073975 (+ 8805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.074529 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.075566 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:29.075772 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:29.075825 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:29.077437 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:29.077438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:29.077439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:29.077444 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:29.078946 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.079352 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.079353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.079353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.079401 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:29.079622 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:29.079622 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:29.079623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:29.079714 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:29.079798 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:29.079935 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:54:29.079982 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:29.082078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20588
W20260925 14:54:29.086442 20573 connection.cc:535] client connection to 127.0.0.1:43105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43105): Connection reset by peer (error 104)
2026-09-25T14:54:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:29 dist-test-slave-gpc1 krb5kdc[20687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:29 dist-test-slave-gpc1 krb5kdc[20687](info): set up 2 sockets
Sep 25 14:54:29 dist-test-slave-gpc1 krb5kdc[20687](info): commencing operation
krb5kdc: starting...
W20260925 14:54:30.118319 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:30 dist-test-slave-gpc1 krb5kdc[20687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:30.234112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:30.313374 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:30.313583 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:30.313606 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:30.314821 20700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:30.314862 20700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:30.314877 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:30.314888 20700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:30.314929 20700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:30.316433 20700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:30.316684 20700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:30.316917 20700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:30.321717 20705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:30.321764 20706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:30.321729 20708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:30.322017 20700 server_base.cc:1034] running on GCE node
I20260925 14:54:30.322280 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:30.322594 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:30.323740 20700 hybrid_clock.cc:648] HybridClock initialized: now 1790348070323736 us; error 36 us; skew 500 ppm
Sep 25 14:54:30 dist-test-slave-gpc1 krb5kdc[20687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:30.329905 20700 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:30.330438 20700 webserver.cc:460] Webserver started at http://127.0.0.1:44693/ using document root <none> and password file <none>
I20260925 14:54:30.330675 20700 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:30.330744 20700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:30.330899 20700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:30.331949 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "c2ad42eb06264e9ea235e70f99df51a0"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-gpc1"
server_key: "d184caf44ad96541680ab05136a44066"
server_key_iv: "047ab5628b051e2c32a034008b2b4f99"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.332306 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2ad42eb06264e9ea235e70f99df51a0"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-gpc1"
server_key: "d184caf44ad96541680ab05136a44066"
server_key_iv: "047ab5628b051e2c32a034008b2b4f99"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.333616 20700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:30.334430 20715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:30.334627 20700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:30.334764 20700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "c2ad42eb06264e9ea235e70f99df51a0"
format_stamp: "Formatted at 2026-09-25 14:54:30 on dist-test-slave-gpc1"
server_key: "d184caf44ad96541680ab05136a44066"
server_key_iv: "047ab5628b051e2c32a034008b2b4f99"
server_key_version: "encryptionkey@0"
I20260925 14:54:30.334851 20700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:30.352124 20700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:30.354547 20700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:30.354707 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:30.358731 20700 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38945
I20260925 14:54:30.358778 20767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38945 every 8 connection(s)
I20260925 14:54:30.359155 20700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:30.359987 20768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:30.362190 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0: Bootstrap starting.
I20260925 14:54:30.362783 20768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:30.363070 20768 log.cc:824] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:30.363739 20768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0: No bootstrap required, opened a new log
I20260925 14:54:30.365268 20768 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } }
I20260925 14:54:30.365402 20768 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:30.365424 20768 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ad42eb06264e9ea235e70f99df51a0, State: Initialized, Role: FOLLOWER
I20260925 14:54:30.365592 20768 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } }
I20260925 14:54:30.365653 20768 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:30.365674 20768 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:30.365739 20768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:30.366264 20768 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } }
I20260925 14:54:30.366406 20768 leader_election.cc:302] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ad42eb06264e9ea235e70f99df51a0; no voters:
I20260925 14:54:30.366596 20768 leader_election.cc:288] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:30.366645 20771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:30.366840 20771 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [term 1 LEADER]: Becoming Leader. State: Replica: c2ad42eb06264e9ea235e70f99df51a0, State: Running, Role: LEADER
I20260925 14:54:30.366848 20768 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:30.367010 20771 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } }
I20260925 14:54:30.367419 20773 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ad42eb06264e9ea235e70f99df51a0. Latest consensus state: current_term: 1 leader_uuid: "c2ad42eb06264e9ea235e70f99df51a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } } }
I20260925 14:54:30.367556 20772 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2ad42eb06264e9ea235e70f99df51a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ad42eb06264e9ea235e70f99df51a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38945 } } }
I20260925 14:54:30.367626 20772 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:30.367548 20773 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:30.367872 20780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:30.368428 20780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:30.368656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20700
I20260925 14:54:30.368765 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:30.368968 392 external_mini_cluster.cc:1428] Setting key fbaee0de60f34f6b42209a7b1c8e6a4c
I20260925 14:54:30.369908 20780 catalog_manager.cc:1347] Generated new cluster ID: ccd1081257c04c839154c1af32fea5a9
I20260925 14:54:30.369961 20780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:30.380157 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.369408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35847 (local address 127.0.0.1:38945)
0925 14:54:30.369555 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.369562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.374618 (+ 5056us) server_negotiation.cc:407] Connection header received
0925 14:54:30.374739 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.374747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.374800 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.374938 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:30.375280 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.376637 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.377775 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.378398 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.378506 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.379260 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:30.379270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:30.379270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:30.379526 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:30.379564 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:30.379577 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:30.379677 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.379902 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.379930 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260925 14:54:30.381047 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:30.382951 20780 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:30.383436 20780 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:30.389039 20780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ad42eb06264e9ea235e70f99df51a0: Generated new TSK 0
I20260925 14:54:30.389209 20780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:30.392659 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:30.389002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56081 (local address 127.0.0.1:38945)
0925 14:54:30.389042 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:30.389043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:30.389327 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:54:30.389398 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:30.389399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:30.389411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:30.389503 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:30.389690 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.390488 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.391279 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:30.391525 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:30.391609 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:30.392393 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:30.392395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:30.392395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:30.392399 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:30.392416 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:30.392425 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:30.392494 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:30.392571 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:54:30.392588 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:30.394491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20700
W20260925 14:54:30.398849 20685 connection.cc:535] client connection to 127.0.0.1:38945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38945): Connection reset by peer (error 104)
2026-09-25T14:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:30 dist-test-slave-gpc1 krb5kdc[20799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:30 dist-test-slave-gpc1 krb5kdc[20799](info): set up 2 sockets
Sep 25 14:54:30 dist-test-slave-gpc1 krb5kdc[20799](info): commencing operation
krb5kdc: starting...
W20260925 14:54:31.418494 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:31 dist-test-slave-gpc1 krb5kdc[20799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:31.533000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:31.613440 20812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:31.613641 20812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:31.613664 20812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:31.614784 20812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:31.614822 20812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:31.614836 20812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:31.614848 20812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:31.614861 20812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:31.616343 20812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39609
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:31.616539 20812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:31.616705 20812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:31.621673 20820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:31.621670 20818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:31.621733 20812 server_base.cc:1034] running on GCE node
W20260925 14:54:31.621670 20817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:31.622143 20812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:31.622465 20812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:31.623610 20812 hybrid_clock.cc:648] HybridClock initialized: now 1790348071623596 us; error 30 us; skew 500 ppm
Sep 25 14:54:31 dist-test-slave-gpc1 krb5kdc[20799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:31.626160 20812 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:31.626694 20812 webserver.cc:460] Webserver started at http://127.0.0.1:41427/ using document root <none> and password file <none>
I20260925 14:54:31.626950 20812 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:31.626996 20812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:31.627162 20812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:31.628208 20812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "1be439c90ed249bea9e61c69bec112b9"
format_stamp: "Formatted at 2026-09-25 14:54:31 on dist-test-slave-gpc1"
server_key: "4e59e2616946ce2c682dece4dafc4d73"
server_key_iv: "6a435b2dbd83049e7518210f494776e1"
server_key_version: "encryptionkey@0"
I20260925 14:54:31.628568 20812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "1be439c90ed249bea9e61c69bec112b9"
format_stamp: "Formatted at 2026-09-25 14:54:31 on dist-test-slave-gpc1"
server_key: "4e59e2616946ce2c682dece4dafc4d73"
server_key_iv: "6a435b2dbd83049e7518210f494776e1"
server_key_version: "encryptionkey@0"
I20260925 14:54:31.629904 20812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:31.630797 20827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:31.630968 20812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:31.631065 20812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "1be439c90ed249bea9e61c69bec112b9"
format_stamp: "Formatted at 2026-09-25 14:54:31 on dist-test-slave-gpc1"
server_key: "4e59e2616946ce2c682dece4dafc4d73"
server_key_iv: "6a435b2dbd83049e7518210f494776e1"
server_key_version: "encryptionkey@0"
I20260925 14:54:31.631182 20812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:31.634706 20812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:31.636848 20812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:31.637022 20812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:31.641103 20812 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39609
I20260925 14:54:31.641130 20879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39609 every 8 connection(s)
I20260925 14:54:31.641474 20812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:31.642489 20880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:31.644695 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9: Bootstrap starting.
I20260925 14:54:31.645215 20880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:31.645457 20880 log.cc:824] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:31.646070 20880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9: No bootstrap required, opened a new log
I20260925 14:54:31.647789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20812
I20260925 14:54:31.647927 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:31.647790 20880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260925 14:54:31.647967 20880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:31.648005 20880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be439c90ed249bea9e61c69bec112b9, State: Initialized, Role: FOLLOWER
I20260925 14:54:31.648135 392 external_mini_cluster.cc:1428] Setting key 6473c84b436ce4064207c6cef0d66759
I20260925 14:54:31.648161 20880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260925 14:54:31.648247 20880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:31.648330 20880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:31.648373 20880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:31.649477 20880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260925 14:54:31.649600 20880 leader_election.cc:302] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be439c90ed249bea9e61c69bec112b9; no voters:
I20260925 14:54:31.649760 20880 leader_election.cc:288] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:31.649823 20885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:31.649969 20885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1be439c90ed249bea9e61c69bec112b9, State: Running, Role: LEADER
I20260925 14:54:31.650022 20880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:31.650087 20885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } }
I20260925 14:54:31.650478 20886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be439c90ed249bea9e61c69bec112b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } } }
I20260925 14:54:31.650482 20887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be439c90ed249bea9e61c69bec112b9. Latest consensus state: current_term: 1 leader_uuid: "1be439c90ed249bea9e61c69bec112b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be439c90ed249bea9e61c69bec112b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39609 } } }
I20260925 14:54:31.650538 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:31.648664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51607 (local address 127.0.0.1:39609)
0925 14:54:31.648911 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:31.648927 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:31.648974 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:54:31.649072 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:31.649077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:31.649100 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:31.649195 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:31.649923 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:31.649926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:31.649927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:31.649931 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:31.649974 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:31.649988 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:31.650046 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:31.650278 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 14:54:31.650326 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":110,"threads_started":1}
I20260925 14:54:31.650565 20886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:31.650568 20887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:31.651430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:31.651546 20891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:31.652145 20891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:31.654072 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:31.652906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54741 (local address 127.0.0.1:39609)
0925 14:54:31.652947 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:31.652949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:31.653101 (+ 152us) server_negotiation.cc:407] Connection header received
0925 14:54:31.653159 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:31.653160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:31.653179 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:31.653237 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:31.653851 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:31.653852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:31.653853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:31.653857 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:31.653870 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:31.653874 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:31.653938 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:31.654003 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:54:31.654016 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:31.654119 20891 catalog_manager.cc:1347] Generated new cluster ID: 7ebcf74ccdaa4283bdfdbbb2e695277f
I20260925 14:54:31.654165 20891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:31.677959 20891 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:31.678174 20891 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:31.681640 20891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be439c90ed249bea9e61c69bec112b9: Generated new TSK 0
I20260925 14:54:31.681829 20891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:31.685796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20812
2026-09-25T14:54:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:31 dist-test-slave-gpc1 krb5kdc[20911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:31 dist-test-slave-gpc1 krb5kdc[20911](info): set up 2 sockets
Sep 25 14:54:31 dist-test-slave-gpc1 krb5kdc[20911](info): commencing operation
krb5kdc: starting...
W20260925 14:54:32.726318 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:32 dist-test-slave-gpc1 krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:32.839139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:32.921277 20924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:32.921465 20924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:32.921487 20924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:32.922567 20924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:32.922602 20924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:32.922617 20924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:32.922631 20924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:32.922642 20924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:32.924017 20924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:32.924185 20924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:32.924348 20924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:32.928992 20930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:32.929019 20929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:32.929160 20924 server_base.cc:1034] running on GCE node
W20260925 14:54:32.929003 20932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:32.929466 20924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:32.929702 20924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:32.930838 20924 hybrid_clock.cc:648] HybridClock initialized: now 1790348072930811 us; error 42 us; skew 500 ppm
Sep 25 14:54:32 dist-test-slave-gpc1 krb5kdc[20911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:32.933198 20924 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:32.933651 20924 webserver.cc:460] Webserver started at http://127.0.0.1:34965/ using document root <none> and password file <none>
I20260925 14:54:32.933885 20924 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:32.933929 20924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:32.934072 20924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:32.935148 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "2ce373525fec465c9ac6a2e34adc302f"
format_stamp: "Formatted at 2026-09-25 14:54:32 on dist-test-slave-gpc1"
server_key: "7ed27d52ec292a2ba2700eea2d21a979"
server_key_iv: "335946a1d003ff07b2ed0384994e596c"
server_key_version: "encryptionkey@0"
I20260925 14:54:32.935501 20924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "2ce373525fec465c9ac6a2e34adc302f"
format_stamp: "Formatted at 2026-09-25 14:54:32 on dist-test-slave-gpc1"
server_key: "7ed27d52ec292a2ba2700eea2d21a979"
server_key_iv: "335946a1d003ff07b2ed0384994e596c"
server_key_version: "encryptionkey@0"
I20260925 14:54:32.936839 20924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:54:32.937578 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:32.937779 20924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:32.937870 20924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "2ce373525fec465c9ac6a2e34adc302f"
format_stamp: "Formatted at 2026-09-25 14:54:32 on dist-test-slave-gpc1"
server_key: "7ed27d52ec292a2ba2700eea2d21a979"
server_key_iv: "335946a1d003ff07b2ed0384994e596c"
server_key_version: "encryptionkey@0"
I20260925 14:54:32.937943 20924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:32.964088 20924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:32.965950 20924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:32.966102 20924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:32.970193 20924 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40789
I20260925 14:54:32.970252 20991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40789 every 8 connection(s)
I20260925 14:54:32.970614 20924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:32.971426 20992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:32.973574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 20924
I20260925 14:54:32.973712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:32.973846 20992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f: Bootstrap starting.
I20260925 14:54:32.973902 392 external_mini_cluster.cc:1428] Setting key 54f85778c6030001885a24c0070b8353
I20260925 14:54:32.974572 20992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:32.974936 20992 log.cc:824] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:32.975719 20992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f: No bootstrap required, opened a new log
I20260925 14:54:32.977880 20992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260925 14:54:32.978047 20992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:32.978080 20992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce373525fec465c9ac6a2e34adc302f, State: Initialized, Role: FOLLOWER
I20260925 14:54:32.978223 20992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260925 14:54:32.978329 20992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:32.978385 20992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:32.978438 20992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:32.979082 20992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260925 14:54:32.979197 20992 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce373525fec465c9ac6a2e34adc302f; no voters:
I20260925 14:54:32.979351 20992 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:32.979378 20997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:32.979490 20997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [term 1 LEADER]: Becoming Leader. State: Replica: 2ce373525fec465c9ac6a2e34adc302f, State: Running, Role: LEADER
I20260925 14:54:32.979615 20992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:32.979619 20997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260925 14:54:32.980193 20999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce373525fec465c9ac6a2e34adc302f. Latest consensus state: current_term: 1 leader_uuid: "2ce373525fec465c9ac6a2e34adc302f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } } }
I20260925 14:54:32.980309 20999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:32.980518 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce373525fec465c9ac6a2e34adc302f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce373525fec465c9ac6a2e34adc302f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } } }
I20260925 14:54:32.980594 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:32.980957 21004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 14:54:32 dist-test-slave-gpc1 krb5kdc[20911](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:54:32.981554 21004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:32.983160 21004 catalog_manager.cc:1347] Generated new cluster ID: 308c7406302847968bb1d240cff9d6e8
I20260925 14:54:32.983211 21004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:32.985280 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:32.974429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58529 (local address 127.0.0.1:40789)
0925 14:54:32.974615 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:32.974631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:32.974655 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:32.975207 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:32.975216 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:32.975281 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:32.975433 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:32.976200 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:32.977059 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:32.978186 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:32.978691 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:32.978771 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:32.981613 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:32.981617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:32.981618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:32.981631 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:32.983542 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:32.983908 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:32.983910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:32.983911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:32.984001 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:32.984167 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:32.984167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:32.984167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:32.984437 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:32.984901 (+ 464us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:32.984966 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:54:32.985086 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260925 14:54:32.986577 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:32.995050 21004 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:32.995169 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:32.987930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57863 (local address 127.0.0.1:40789)
0925 14:54:32.987960 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:32.987962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:32.988082 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:54:32.988140 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:32.988141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:32.988150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:32.988202 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:32.988808 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:32.989512 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:32.990535 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:32.990839 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:32.990900 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:32.992269 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:32.992270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:32.992271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:32.992278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:32.993816 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:32.994201 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:32.994202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:32.994203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:32.994260 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:32.994486 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:32.994487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:32.994488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:32.994607 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:32.994724 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:32.994854 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:54:32.994982 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:54:32.995711 21004 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:33.004130 21004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ce373525fec465c9ac6a2e34adc302f: Generated new TSK 0
I20260925 14:54:33.004364 21004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:33.006609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 20924
2026-09-25T14:54:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:33 dist-test-slave-gpc1 krb5kdc[21023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:33 dist-test-slave-gpc1 krb5kdc[21023](info): set up 2 sockets
Sep 25 14:54:33 dist-test-slave-gpc1 krb5kdc[21023](info): commencing operation
krb5kdc: starting...
W20260925 14:54:34.046329 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:34 dist-test-slave-gpc1 krb5kdc[21023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:34.158210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:34.237181 21036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:34.237363 21036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:34.237385 21036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:34.238479 21036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:34.238512 21036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:34.238526 21036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:34.238538 21036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:34.238550 21036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:34.239907 21036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:34.240080 21036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:34.240239 21036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:34.244884 21041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:34.244875 21044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:34.245088 21036 server_base.cc:1034] running on GCE node
W20260925 14:54:34.244943 21042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:34.245401 21036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:34.245714 21036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:34.246860 21036 hybrid_clock.cc:648] HybridClock initialized: now 1790348074246841 us; error 37 us; skew 500 ppm
Sep 25 14:54:34 dist-test-slave-gpc1 krb5kdc[21023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:34.249289 21036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:34.249718 21036 webserver.cc:460] Webserver started at http://127.0.0.1:42193/ using document root <none> and password file <none>
I20260925 14:54:34.249908 21036 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:34.249948 21036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:34.250028 21036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:34.251029 21036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e7425e458c8244dbbe20a3aa75d79a7b"
format_stamp: "Formatted at 2026-09-25 14:54:34 on dist-test-slave-gpc1"
server_key: "2151efbee4f58237cdd7cc4eacb3c04d"
server_key_iv: "871a43491ef3646c47660da31555c411"
server_key_version: "encryptionkey@0"
I20260925 14:54:34.251322 21036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7425e458c8244dbbe20a3aa75d79a7b"
format_stamp: "Formatted at 2026-09-25 14:54:34 on dist-test-slave-gpc1"
server_key: "2151efbee4f58237cdd7cc4eacb3c04d"
server_key_iv: "871a43491ef3646c47660da31555c411"
server_key_version: "encryptionkey@0"
I20260925 14:54:34.252535 21036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:34.253182 21051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:34.253355 21036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:34.253454 21036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e7425e458c8244dbbe20a3aa75d79a7b"
format_stamp: "Formatted at 2026-09-25 14:54:34 on dist-test-slave-gpc1"
server_key: "2151efbee4f58237cdd7cc4eacb3c04d"
server_key_iv: "871a43491ef3646c47660da31555c411"
server_key_version: "encryptionkey@0"
I20260925 14:54:34.253535 21036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:34.273895 21036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:34.275928 21036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:34.276090 21036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:34.280202 21036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34519
I20260925 14:54:34.280232 21103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34519 every 8 connection(s)
I20260925 14:54:34.280570 21036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:34.281411 21104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:34.282524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21036
I20260925 14:54:34.282646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:34.282848 392 external_mini_cluster.cc:1428] Setting key 0b7bc594cedfa81de7fde6648699ea67
I20260925 14:54:34.284096 21104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: Bootstrap starting.
I20260925 14:54:34.284776 21104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:34.285055 21104 log.cc:824] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:34.285799 21104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: No bootstrap required, opened a new log
I20260925 14:54:34.288074 21104 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 14:54:34.288256 21104 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:34.288326 21104 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7425e458c8244dbbe20a3aa75d79a7b, State: Initialized, Role: FOLLOWER
I20260925 14:54:34.288496 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:34.283363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59947 (local address 127.0.0.1:34519)
0925 14:54:34.283544 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:34.283551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:34.283575 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:34.283995 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:34.284002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:34.284047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:34.284161 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:34.284399 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:34.285492 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:34.286625 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:34.287035 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:34.287122 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:34.287797 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:34.287801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:34.287802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:34.287956 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:34.287986 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:34.287996 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:34.288046 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:34.288154 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:54:34.288227 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260925 14:54:34.288496 21104 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 14:54:34.288602 21104 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:34.288643 21104 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:34.288697 21104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:34.289281 21104 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 14:54:34.289369 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:34.289402 21104 leader_election.cc:302] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7425e458c8244dbbe20a3aa75d79a7b; no voters:
I20260925 14:54:34.289554 21104 leader_election.cc:288] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:34.289588 21109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:34.289738 21109 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [term 1 LEADER]: Becoming Leader. State: Replica: e7425e458c8244dbbe20a3aa75d79a7b, State: Running, Role: LEADER
I20260925 14:54:34.289764 21104 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:34.289810 21109 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } }
I20260925 14:54:34.290143 21109 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7425e458c8244dbbe20a3aa75d79a7b. Latest consensus state: current_term: 1 leader_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260925 14:54:34.290205 21109 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:34.291412 21110 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7425e458c8244dbbe20a3aa75d79a7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34519 } } }
I20260925 14:54:34.291678 21110 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b [sys.catalog]: This master's current role is: LEADER
W20260925 14:54:34.291889 21129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:34.291944 21129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:34.292024 21115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:34.292245 21115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:34.293833 21115 catalog_manager.cc:1347] Generated new cluster ID: a336fb4d3ea246ea9d6e20d3d34146ee
I20260925 14:54:34.294078 21115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:34.294373 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:34.291013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60605 (local address 127.0.0.1:34519)
0925 14:54:34.291050 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:34.291052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:34.291093 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:54:34.291180 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:34.291181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:34.291190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:34.291251 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:34.291435 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:34.292135 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:34.293089 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:34.293369 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:34.293443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:34.294087 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:34.294088 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:34.294088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:34.294092 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:34.294107 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:34.294111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:34.294167 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:34.294253 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:54:34.294274 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:34.313560 21115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:34.314098 21115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:34.321655 21115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7425e458c8244dbbe20a3aa75d79a7b: Generated new TSK 0
I20260925 14:54:34.321835 21115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:34.326622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21036
2026-09-25T14:54:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:34 dist-test-slave-gpc1 krb5kdc[21135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:34 dist-test-slave-gpc1 krb5kdc[21135](info): set up 2 sockets
Sep 25 14:54:34 dist-test-slave-gpc1 krb5kdc[21135](info): commencing operation
krb5kdc: starting...
W20260925 14:54:35.366261 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:35 dist-test-slave-gpc1 krb5kdc[21135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:35.476958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:35.556129 21148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:35.556381 21148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:35.556423 21148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:35.557605 21148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:35.557677 21148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:35.557703 21148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:35.557731 21148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:35.557763 21148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:35.559357 21148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:35.559638 21148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:35.559865 21148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:35.564807 21153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:35.564765 21156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:35.564769 21154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:35.564896 21148 server_base.cc:1034] running on GCE node
I20260925 14:54:35.565234 21148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:35.565526 21148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:35.566666 21148 hybrid_clock.cc:648] HybridClock initialized: now 1790348075566655 us; error 27 us; skew 500 ppm
Sep 25 14:54:35 dist-test-slave-gpc1 krb5kdc[21135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:35.572548 21148 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:35.572991 21148 webserver.cc:460] Webserver started at http://127.0.0.1:42219/ using document root <none> and password file <none>
I20260925 14:54:35.573241 21148 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:35.573312 21148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:35.573462 21148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:35.574584 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "4e3b575b1cdb4da7b89858b127893ebe"
format_stamp: "Formatted at 2026-09-25 14:54:35 on dist-test-slave-gpc1"
server_key: "d327ea0561258f15a411933010477832"
server_key_iv: "cfaaa7aaab42f496d9dfe6688772d630"
server_key_version: "encryptionkey@0"
I20260925 14:54:35.574944 21148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e3b575b1cdb4da7b89858b127893ebe"
format_stamp: "Formatted at 2026-09-25 14:54:35 on dist-test-slave-gpc1"
server_key: "d327ea0561258f15a411933010477832"
server_key_iv: "cfaaa7aaab42f496d9dfe6688772d630"
server_key_version: "encryptionkey@0"
I20260925 14:54:35.576225 21148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:35.576995 21163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:35.577188 21148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:35.577294 21148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "4e3b575b1cdb4da7b89858b127893ebe"
format_stamp: "Formatted at 2026-09-25 14:54:35 on dist-test-slave-gpc1"
server_key: "d327ea0561258f15a411933010477832"
server_key_iv: "cfaaa7aaab42f496d9dfe6688772d630"
server_key_version: "encryptionkey@0"
I20260925 14:54:35.577368 21148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:35.580921 21148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:35.582835 21148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:35.582988 21148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:35.586654 21148 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34977
I20260925 14:54:35.586704 21215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34977 every 8 connection(s)
I20260925 14:54:35.587044 21148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:35.588008 21216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:35.590111 21216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe: Bootstrap starting.
I20260925 14:54:35.590691 21216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:35.590916 21216 log.cc:824] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:35.591476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21148
I20260925 14:54:35.591519 21216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe: No bootstrap required, opened a new log
I20260925 14:54:35.591601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:35.591796 392 external_mini_cluster.cc:1428] Setting key f90dc02f4b0fa53f8e3bb91a3a6d5218
I20260925 14:54:35.593421 21216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260925 14:54:35.593583 21216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:35.593622 21216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3b575b1cdb4da7b89858b127893ebe, State: Initialized, Role: FOLLOWER
I20260925 14:54:35.593811 21216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260925 14:54:35.593914 21216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:35.593937 21216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:35.594008 21216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:35.594774 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:35.592292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43007 (local address 127.0.0.1:34977)
0925 14:54:35.592460 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:35.592472 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:35.592496 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:35.593410 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:35.593417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:35.593444 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:35.593555 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:35.594263 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:35.594266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:35.594266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:35.594270 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:35.594305 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:35.594316 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:35.594445 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:35.594520 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:54:35.594550 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 14:54:35.595181 21216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260925 14:54:35.595304 21216 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3b575b1cdb4da7b89858b127893ebe; no voters:
I20260925 14:54:35.595449 21216 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:35.595474 21221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:35.595577 21221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [term 1 LEADER]: Becoming Leader. State: Replica: 4e3b575b1cdb4da7b89858b127893ebe, State: Running, Role: LEADER
I20260925 14:54:35.595661 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:35.595697 21216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:35.595685 21221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } }
I20260925 14:54:35.595980 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3b575b1cdb4da7b89858b127893ebe. Latest consensus state: current_term: 1 leader_uuid: "4e3b575b1cdb4da7b89858b127893ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260925 14:54:35.595989 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3b575b1cdb4da7b89858b127893ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3b575b1cdb4da7b89858b127893ebe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34977 } } }
I20260925 14:54:35.596067 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:35.596089 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:35.596294 21227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:35.596791 21227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:35.598752 21227 catalog_manager.cc:1347] Generated new cluster ID: 04c2320ebd304bdb895c67e669462b9f
I20260925 14:54:35.598809 21227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:35.599488 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:35.597326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33095 (local address 127.0.0.1:34977)
0925 14:54:35.597357 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:35.597359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:35.598517 (+ 1158us) server_negotiation.cc:407] Connection header received
0925 14:54:35.598571 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:35.598572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:35.598576 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:35.598648 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:35.599241 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:35.599242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:35.599243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:35.599247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:35.599259 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:35.599264 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:35.599295 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:35.599414 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 14:54:35.599427 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:54:35.609684 21227 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:35.609869 21227 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:35.612524 21227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e3b575b1cdb4da7b89858b127893ebe: Generated new TSK 0
I20260925 14:54:35.612700 21227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:35.619726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21148
2026-09-25T14:54:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:35 dist-test-slave-gpc1 krb5kdc[21247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:35 dist-test-slave-gpc1 krb5kdc[21247](info): set up 2 sockets
Sep 25 14:54:35 dist-test-slave-gpc1 krb5kdc[21247](info): commencing operation
krb5kdc: starting...
W20260925 14:54:36.662629 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:36 dist-test-slave-gpc1 krb5kdc[21247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:36.776628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:36.866228 21260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:36.866508 21260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:36.866554 21260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:36.867883 21260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:36.867960 21260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:36.867986 21260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:36.868010 21260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:36.868041 21260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:36.869650 21260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:36.869952 21260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:36.870208 21260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:36.875159 21268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:36.875204 21266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:36.875415 21260 server_base.cc:1034] running on GCE node
W20260925 14:54:36.875159 21265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:36.875694 21260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:36.876016 21260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:36.877192 21260 hybrid_clock.cc:648] HybridClock initialized: now 1790348076877178 us; error 32 us; skew 500 ppm
Sep 25 14:54:36 dist-test-slave-gpc1 krb5kdc[21247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:36.879824 21260 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:36.880249 21260 webserver.cc:460] Webserver started at http://127.0.0.1:37561/ using document root <none> and password file <none>
I20260925 14:54:36.880496 21260 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:36.880594 21260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:36.880725 21260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:36.881803 21260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "3f6dfc8a457a43b286cca59bcbefd3dc"
format_stamp: "Formatted at 2026-09-25 14:54:36 on dist-test-slave-gpc1"
server_key: "1642a5e08a2db4c6bb926a75bca6c40f"
server_key_iv: "083ac61a0f145089aacb77ee0910768e"
server_key_version: "encryptionkey@0"
I20260925 14:54:36.882174 21260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f6dfc8a457a43b286cca59bcbefd3dc"
format_stamp: "Formatted at 2026-09-25 14:54:36 on dist-test-slave-gpc1"
server_key: "1642a5e08a2db4c6bb926a75bca6c40f"
server_key_iv: "083ac61a0f145089aacb77ee0910768e"
server_key_version: "encryptionkey@0"
I20260925 14:54:36.883585 21260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:36.884347 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:36.884545 21260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:54:36.884644 21260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "3f6dfc8a457a43b286cca59bcbefd3dc"
format_stamp: "Formatted at 2026-09-25 14:54:36 on dist-test-slave-gpc1"
server_key: "1642a5e08a2db4c6bb926a75bca6c40f"
server_key_iv: "083ac61a0f145089aacb77ee0910768e"
server_key_version: "encryptionkey@0"
I20260925 14:54:36.884727 21260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:36.901050 21260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:36.903230 21260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:36.903400 21260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:36.907428 21260 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45903
I20260925 14:54:36.907483 21327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45903 every 8 connection(s)
I20260925 14:54:36.907860 21260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:36.908710 21328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:36.911051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21260
I20260925 14:54:36.911166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:36.911347 392 external_mini_cluster.cc:1428] Setting key 3c688fcaa0079eec91b8405f968cee25
I20260925 14:54:36.911544 21328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc: Bootstrap starting.
I20260925 14:54:36.912274 21328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:36.912582 21328 log.cc:824] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:36.913312 21328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc: No bootstrap required, opened a new log
I20260925 14:54:36.915223 21328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } }
I20260925 14:54:36.915391 21328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:36.915444 21328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6dfc8a457a43b286cca59bcbefd3dc, State: Initialized, Role: FOLLOWER
I20260925 14:54:36.915603 21328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } }
I20260925 14:54:36.915704 21328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:36.915768 21328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:36.915824 21328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:36.916380 21328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } }
I20260925 14:54:36.916507 21328 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6dfc8a457a43b286cca59bcbefd3dc; no voters:
I20260925 14:54:36.916723 21328 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:36.916750 21333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:36.916882 21333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [term 1 LEADER]: Becoming Leader. State: Replica: 3f6dfc8a457a43b286cca59bcbefd3dc, State: Running, Role: LEADER
I20260925 14:54:36.916990 21328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:36.917025 21333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } }
I20260925 14:54:36.917457 21333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6dfc8a457a43b286cca59bcbefd3dc. Latest consensus state: current_term: 1 leader_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } } }
I20260925 14:54:36.917721 21333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [sys.catalog]: This master's current role is: LEADER
Sep 25 14:54:36 dist-test-slave-gpc1 krb5kdc[21247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:54:36.917639 21335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6dfc8a457a43b286cca59bcbefd3dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45903 } } }
I20260925 14:54:36.918126 21335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:36.918298 21342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:36.918856 21342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:36.920585 21342 catalog_manager.cc:1347] Generated new cluster ID: ba3e0a96d35446539b8219b1d9fe5eb0
I20260925 14:54:36.920650 21342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:36.922654 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:36.911891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:58669 (local address 127.0.0.1:45903)
0925 14:54:36.912095 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:36.912103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:36.912127 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:36.912589 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:36.912596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:36.912645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:36.912820 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:36.913456 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:36.914389 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:36.915191 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:36.915674 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:36.915800 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:36.918766 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:36.918770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:36.918771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:36.918784 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:36.920918 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:36.921394 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:36.921397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:36.921397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:36.921502 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:36.921750 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:36.921751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:36.921751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:36.922030 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:36.922194 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:36.922320 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:54:36.922441 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260925 14:54:36.923648 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:36.933718 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:36.925373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:34883 (local address 127.0.0.1:45903)
0925 14:54:36.925429 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:36.925431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:36.926518 (+ 1087us) server_negotiation.cc:407] Connection header received
0925 14:54:36.926568 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:36.926569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:36.926579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:36.926639 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:36.927273 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:36.928035 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:36.929026 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:36.929366 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:36.929438 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:36.930877 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:36.930878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:36.930879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:36.930885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:36.932461 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:36.932881 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:36.932882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:36.932883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:36.932941 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:36.933182 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:36.933183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:36.933183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:36.933304 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:36.933410 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:36.933566 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:54:36.933630 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:54:36.935344 21342 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:36.935942 21342 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:36.943675 21342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6dfc8a457a43b286cca59bcbefd3dc: Generated new TSK 0
I20260925 14:54:36.943867 21342 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:36.947432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21260
W20260925 14:54:36.951508 21245 connection.cc:535] client connection to 127.0.0.1:45903 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45903): Connection reset by peer (error 104)
2026-09-25T14:54:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:37 dist-test-slave-gpc1 krb5kdc[21359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:37 dist-test-slave-gpc1 krb5kdc[21359](info): set up 2 sockets
Sep 25 14:54:37 dist-test-slave-gpc1 krb5kdc[21359](info): commencing operation
krb5kdc: starting...
W20260925 14:54:37.998567 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:38 dist-test-slave-gpc1 krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:38.112422 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:38.192273 21372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:38.192483 21372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:38.192507 21372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:38.193682 21372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:38.193725 21372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:38.193739 21372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:38.193753 21372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:38.193764 21372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:38.195256 21372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:38.195513 21372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:38.195744 21372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:38.200554 21377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:38.200553 21380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:38.200641 21378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:38.200870 21372 server_base.cc:1034] running on GCE node
I20260925 14:54:38.201079 21372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:38.201363 21372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:38.202502 21372 hybrid_clock.cc:648] HybridClock initialized: now 1790348078202488 us; error 28 us; skew 500 ppm
Sep 25 14:54:38 dist-test-slave-gpc1 krb5kdc[21359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:38.204975 21372 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:38.205425 21372 webserver.cc:460] Webserver started at http://127.0.0.1:35805/ using document root <none> and password file <none>
I20260925 14:54:38.205605 21372 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:38.205643 21372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:38.205737 21372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:38.206831 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "9303506955b740d39fffc03fac3c0729"
format_stamp: "Formatted at 2026-09-25 14:54:38 on dist-test-slave-gpc1"
server_key: "157b60f8f7980e4e5b50887c5ebf3a10"
server_key_iv: "55498a2cb3712acc47a37524a6f590ea"
server_key_version: "encryptionkey@0"
I20260925 14:54:38.207149 21372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "9303506955b740d39fffc03fac3c0729"
format_stamp: "Formatted at 2026-09-25 14:54:38 on dist-test-slave-gpc1"
server_key: "157b60f8f7980e4e5b50887c5ebf3a10"
server_key_iv: "55498a2cb3712acc47a37524a6f590ea"
server_key_version: "encryptionkey@0"
I20260925 14:54:38.208396 21372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:38.209081 21387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:38.209304 21372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:38.209380 21372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "9303506955b740d39fffc03fac3c0729"
format_stamp: "Formatted at 2026-09-25 14:54:38 on dist-test-slave-gpc1"
server_key: "157b60f8f7980e4e5b50887c5ebf3a10"
server_key_iv: "55498a2cb3712acc47a37524a6f590ea"
server_key_version: "encryptionkey@0"
I20260925 14:54:38.209463 21372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:38.217972 21372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:38.219993 21372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:38.220149 21372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:38.224123 21372 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40179
I20260925 14:54:38.224175 21439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40179 every 8 connection(s)
I20260925 14:54:38.224509 21372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:38.225407 21440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:38.226907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21372
I20260925 14:54:38.227032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:38.227252 392 external_mini_cluster.cc:1428] Setting key 3f514ad2ddb22464717aa2567495103a
I20260925 14:54:38.228286 21440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729: Bootstrap starting.
I20260925 14:54:38.228987 21440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:38.229300 21440 log.cc:824] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:38.230046 21440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729: No bootstrap required, opened a new log
I20260925 14:54:38.232493 21440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260925 14:54:38.232710 21440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:38.232795 21440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9303506955b740d39fffc03fac3c0729, State: Initialized, Role: FOLLOWER
I20260925 14:54:38.232954 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:38.227805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:35425 (local address 127.0.0.1:40179)
0925 14:54:38.227971 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:38.227979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:38.228003 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:54:38.228442 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:38.228449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:38.228496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:38.228607 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:38.228896 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:38.229871 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:38.230995 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:38.231391 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:38.231466 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:38.232210 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:38.232213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:38.232213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:38.232366 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:38.232408 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:38.232418 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:38.232477 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:38.232623 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:54:38.232667 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260925 14:54:38.232941 21440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260925 14:54:38.233047 21440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:38.233101 21440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:38.233156 21440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:38.233753 21440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260925 14:54:38.233872 21440 leader_election.cc:302] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9303506955b740d39fffc03fac3c0729; no voters:
I20260925 14:54:38.233924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:38.234068 21440 leader_election.cc:288] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:38.234113 21445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:38.234242 21445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [term 1 LEADER]: Becoming Leader. State: Replica: 9303506955b740d39fffc03fac3c0729, State: Running, Role: LEADER
I20260925 14:54:38.234359 21440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:38.234404 21445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260925 14:54:38.234750 21445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9303506955b740d39fffc03fac3c0729. Latest consensus state: current_term: 1 leader_uuid: "9303506955b740d39fffc03fac3c0729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } } }
I20260925 14:54:38.234759 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9303506955b740d39fffc03fac3c0729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9303506955b740d39fffc03fac3c0729" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } } }
I20260925 14:54:38.234865 21445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:38.234879 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:38.235118 21451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:38.235692 21451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:38.239545 21451 catalog_manager.cc:1347] Generated new cluster ID: 6f18bfd366b446f196a717b5678891c2
I20260925 14:54:38.239605 21451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:38.240020 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:38.236232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:58731 (local address 127.0.0.1:40179)
0925 14:54:38.236269 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:38.236271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:38.236516 (+ 245us) server_negotiation.cc:407] Connection header received
0925 14:54:38.236568 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:38.236569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:38.236579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:38.236636 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:38.236759 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:38.237230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:38.238290 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:38.238679 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:38.238795 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:38.239726 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:38.239728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:38.239728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:38.239733 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:38.239749 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:38.239754 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:38.239839 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:38.239911 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:54:38.239921 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:38.251281 21451 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:38.251986 21451 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:38.263151 21451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9303506955b740d39fffc03fac3c0729: Generated new TSK 0
I20260925 14:54:38.263357 21451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:38.266121 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21372
W20260925 14:54:38.270645 21357 connection.cc:535] client connection to 127.0.0.1:40179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40179): Connection reset by peer (error 104)
2026-09-25T14:54:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:38 dist-test-slave-gpc1 krb5kdc[21471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:38 dist-test-slave-gpc1 krb5kdc[21471](info): set up 2 sockets
Sep 25 14:54:38 dist-test-slave-gpc1 krb5kdc[21471](info): commencing operation
krb5kdc: starting...
W20260925 14:54:39.302469 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:39 dist-test-slave-gpc1 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:39.413404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:39.492347 21484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:39.492594 21484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:39.492631 21484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:39.493754 21484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:39.493825 21484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:39.493849 21484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:39.493871 21484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:39.493899 21484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:39.495359 21484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:39.495640 21484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:39.495869 21484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:39.500717 21489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:39.500723 21492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:39.500929 21484 server_base.cc:1034] running on GCE node
W20260925 14:54:39.500931 21490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:39.501219 21484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:39.501513 21484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:39.502671 21484 hybrid_clock.cc:648] HybridClock initialized: now 1790348079502663 us; error 50 us; skew 500 ppm
Sep 25 14:54:39 dist-test-slave-gpc1 krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:39.505131 21484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:39.505605 21484 webserver.cc:460] Webserver started at http://127.0.0.1:36937/ using document root <none> and password file <none>
I20260925 14:54:39.505847 21484 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:39.505916 21484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:39.506060 21484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:39.507193 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "97431b0089f74f82b4cc449a12cfb4ed"
format_stamp: "Formatted at 2026-09-25 14:54:39 on dist-test-slave-gpc1"
server_key: "da7da734510adc3f65d95befb0862738"
server_key_iv: "8fd6eb58dfeb70a6f4535e97ca00b0d5"
server_key_version: "encryptionkey@0"
I20260925 14:54:39.507555 21484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "97431b0089f74f82b4cc449a12cfb4ed"
format_stamp: "Formatted at 2026-09-25 14:54:39 on dist-test-slave-gpc1"
server_key: "da7da734510adc3f65d95befb0862738"
server_key_iv: "8fd6eb58dfeb70a6f4535e97ca00b0d5"
server_key_version: "encryptionkey@0"
I20260925 14:54:39.508857 21484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:54:39.509560 21499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:39.509728 21484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:39.509833 21484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "97431b0089f74f82b4cc449a12cfb4ed"
format_stamp: "Formatted at 2026-09-25 14:54:39 on dist-test-slave-gpc1"
server_key: "da7da734510adc3f65d95befb0862738"
server_key_iv: "8fd6eb58dfeb70a6f4535e97ca00b0d5"
server_key_version: "encryptionkey@0"
I20260925 14:54:39.509908 21484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:39.513234 21484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:39.515594 21484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:39.515760 21484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:39.519814 21484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33821
I20260925 14:54:39.519851 21551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33821 every 8 connection(s)
I20260925 14:54:39.520166 21484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:39.521095 21552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:39.523339 21552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed: Bootstrap starting.
I20260925 14:54:39.523855 21552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:39.524083 21552 log.cc:824] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:39.524699 21552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed: No bootstrap required, opened a new log
I20260925 14:54:39.526187 21552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260925 14:54:39.526330 21552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:39.526441 21552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97431b0089f74f82b4cc449a12cfb4ed, State: Initialized, Role: FOLLOWER
I20260925 14:54:39.526551 21552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260925 14:54:39.526607 21552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:39.526628 21552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:39.526654 21552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:39.527141 21552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260925 14:54:39.527258 21552 leader_election.cc:302] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97431b0089f74f82b4cc449a12cfb4ed; no voters:
I20260925 14:54:39.527426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21484
I20260925 14:54:39.527444 21552 leader_election.cc:288] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:39.527513 21555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:39.527545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:39.527729 392 external_mini_cluster.cc:1428] Setting key f0578d1e7b20f6154ff371c59aac0d12
I20260925 14:54:39.527706 21552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:39.527705 21555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [term 1 LEADER]: Becoming Leader. State: Replica: 97431b0089f74f82b4cc449a12cfb4ed, State: Running, Role: LEADER
I20260925 14:54:39.527841 21555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } }
I20260925 14:54:39.528168 21556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97431b0089f74f82b4cc449a12cfb4ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } } }
I20260925 14:54:39.528316 21556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:39.528323 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97431b0089f74f82b4cc449a12cfb4ed. Latest consensus state: current_term: 1 leader_uuid: "97431b0089f74f82b4cc449a12cfb4ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97431b0089f74f82b4cc449a12cfb4ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33821 } } }
I20260925 14:54:39.528717 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:39.528774 21560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:39.529306 21560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:39.530915 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:39.528441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:40713 (local address 127.0.0.1:33821)
0925 14:54:39.528729 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:39.528741 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:39.528962 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:54:39.529095 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:39.529102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:39.529135 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:39.529576 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:39.530455 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:39.530459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:39.530459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:39.530463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:39.530509 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:39.530526 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:39.530598 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:39.530693 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:39.530739 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
I20260925 14:54:39.531155 21560 catalog_manager.cc:1347] Generated new cluster ID: 2a1d12acb86d460db7c31917eed010b7
I20260925 14:54:39.531199 21560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:39.531802 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:39.543572 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:39.534664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:54567 (local address 127.0.0.1:33821)
0925 14:54:39.534697 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:39.534699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:39.542547 (+ 7848us) server_negotiation.cc:407] Connection header received
0925 14:54:39.542603 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:39.542604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:39.542610 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:39.542678 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:39.543332 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:39.543333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:39.543334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:39.543338 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:39.543355 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:39.543360 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:39.543420 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:39.543472 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:54:39.543485 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:54:39.572212 21560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:39.572423 21560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:39.575078 21560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97431b0089f74f82b4cc449a12cfb4ed: Generated new TSK 0
I20260925 14:54:39.575292 21560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:39.584684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21484
2026-09-25T14:54:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:39 dist-test-slave-gpc1 krb5kdc[21583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:39 dist-test-slave-gpc1 krb5kdc[21583](info): set up 2 sockets
Sep 25 14:54:39 dist-test-slave-gpc1 krb5kdc[21583](info): commencing operation
krb5kdc: starting...
W20260925 14:54:40.622524 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:40.735208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:40.818435 21596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:40.818676 21596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:40.818727 21596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:40.819890 21596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:40.819957 21596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:40.819983 21596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:40.820014 21596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:40.820045 21596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:40.821738 21596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:40.822013 21596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:40.822244 21596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:40.828529 21602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:40.828547 21604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:40.828516 21601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:40.829150 21596 server_base.cc:1034] running on GCE node
I20260925 14:54:40.829401 21596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:40.829749 21596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:40.830931 21596 hybrid_clock.cc:648] HybridClock initialized: now 1790348080830908 us; error 40 us; skew 500 ppm
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:40.839720 21596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:40.840134 21596 webserver.cc:460] Webserver started at http://127.0.0.1:41893/ using document root <none> and password file <none>
I20260925 14:54:40.840325 21596 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:40.840366 21596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:40.840456 21596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:40.841490 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "e870542d6fe249169c78e6b311302797"
format_stamp: "Formatted at 2026-09-25 14:54:40 on dist-test-slave-gpc1"
server_key: "0c40a560ef670b9b23eeb7a0ed9d28a8"
server_key_iv: "1737995dd2f4d5583f8e6acbd17889b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:40.841789 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "e870542d6fe249169c78e6b311302797"
format_stamp: "Formatted at 2026-09-25 14:54:40 on dist-test-slave-gpc1"
server_key: "0c40a560ef670b9b23eeb7a0ed9d28a8"
server_key_iv: "1737995dd2f4d5583f8e6acbd17889b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:40.843163 21596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:54:40.843876 21611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:40.844096 21596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:40.844197 21596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "e870542d6fe249169c78e6b311302797"
format_stamp: "Formatted at 2026-09-25 14:54:40 on dist-test-slave-gpc1"
server_key: "0c40a560ef670b9b23eeb7a0ed9d28a8"
server_key_iv: "1737995dd2f4d5583f8e6acbd17889b4"
server_key_version: "encryptionkey@0"
I20260925 14:54:40.844252 21596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:40.854739 21596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:40.856422 21596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:40.856526 21596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:40.860241 21596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41917
I20260925 14:54:40.860317 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41917 every 8 connection(s)
I20260925 14:54:40.860638 21596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:40.861474 21664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:40.863739 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797: Bootstrap starting.
I20260925 14:54:40.864291 21664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:40.864576 21664 log.cc:824] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:40.865227 21664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797: No bootstrap required, opened a new log
I20260925 14:54:40.866885 21664 raft_consensus.cc:348] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260925 14:54:40.867040 21664 raft_consensus.cc:374] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:40.867089 21664 raft_consensus.cc:729] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e870542d6fe249169c78e6b311302797, State: Initialized, Role: FOLLOWER
I20260925 14:54:40.867229 21664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260925 14:54:40.867318 21664 raft_consensus.cc:388] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:40.867362 21664 raft_consensus.cc:482] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:40.867413 21664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:40.867908 21664 raft_consensus.cc:504] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260925 14:54:40.868022 21664 leader_election.cc:302] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e870542d6fe249169c78e6b311302797; no voters:
I20260925 14:54:40.868198 21664 leader_election.cc:288] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:40.868242 21667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:40.868427 21667 raft_consensus.cc:686] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [term 1 LEADER]: Becoming Leader. State: Replica: e870542d6fe249169c78e6b311302797, State: Running, Role: LEADER
I20260925 14:54:40.868462 21664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:40.868604 21667 consensus_queue.cc:237] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260925 14:54:40.868969 21668 sys_catalog.cc:455] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e870542d6fe249169c78e6b311302797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260925 14:54:40.868973 21669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e870542d6fe249169c78e6b311302797. Latest consensus state: current_term: 1 leader_uuid: "e870542d6fe249169c78e6b311302797" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e870542d6fe249169c78e6b311302797" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260925 14:54:40.869107 21668 sys_catalog.cc:458] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:40.869128 21669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:40.869326 21671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:40.869624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21596
I20260925 14:54:40.869756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:40.869850 21671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:40.869964 392 external_mini_cluster.cc:1428] Setting key 266a8f4ac54d21b109c49d8ac7b70282
I20260925 14:54:40.871649 21671 catalog_manager.cc:1347] Generated new cluster ID: dccd6d4ad55f4404ab238c555dda2401
I20260925 14:54:40.871711 21671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:54:40.880424 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:40.870487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:35739 (local address 127.0.0.1:41917)
0925 14:54:40.871049 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:40.871058 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:40.871112 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:54:40.871240 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:40.871247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:40.871316 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:40.871452 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:40.872156 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:40.873300 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:40.874063 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:40.874564 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:40.874688 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:40.877025 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:40.877028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:40.877028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:40.877040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:40.878840 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:40.879234 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:40.879237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:40.879237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:40.879341 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:40.879591 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:40.879592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:40.879592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:40.879845 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:40.880031 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:40.880124 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:54:40.880227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":73,"threads_started":1}
I20260925 14:54:40.881495 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:40.883098 21671 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:40.883648 21671 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:40.887516 21671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e870542d6fe249169c78e6b311302797: Generated new TSK 0
I20260925 14:54:40.887703 21671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:40.894476 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:40.882821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:44923 (local address 127.0.0.1:41917)
0925 14:54:40.882861 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:40.882863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:40.883065 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:54:40.883162 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:40.883163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:40.883173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:40.883200 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:40.887893 (+ 4693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:40.888682 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:40.889799 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:40.890160 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:40.890209 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:40.891916 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:40.891917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:40.891917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:40.891924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:40.893282 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:40.893664 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:40.893665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:40.893665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:40.893722 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:40.893952 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:40.893953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:40.893953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:40.894062 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:40.894174 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:40.894290 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:54:40.894381 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:54:40.896345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21596
W20260925 14:54:40.900666 21581 connection.cc:535] client connection to 127.0.0.1:41917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41917): Connection reset by peer (error 104)
2026-09-25T14:54:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21695](info): set up 2 sockets
Sep 25 14:54:40 dist-test-slave-gpc1 krb5kdc[21695](info): commencing operation
krb5kdc: starting...
W20260925 14:54:41.958437 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:42 dist-test-slave-gpc1 krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:42.070142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:42.150395 21708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:42.150619 21708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:42.150641 21708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:42.151734 21708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:42.151768 21708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:42.151782 21708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:42.151795 21708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:42.151808 21708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:42.153272 21708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:42.153461 21708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:42.153614 21708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:42.158432 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:42.158424 21714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:42.158494 21708 server_base.cc:1034] running on GCE node
W20260925 14:54:42.158437 21716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:42.158895 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:42.159211 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:42.160387 21708 hybrid_clock.cc:648] HybridClock initialized: now 1790348082160374 us; error 27 us; skew 500 ppm
Sep 25 14:54:42 dist-test-slave-gpc1 krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:42.162941 21708 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:42.163379 21708 webserver.cc:460] Webserver started at http://127.0.0.1:45235/ using document root <none> and password file <none>
I20260925 14:54:42.163625 21708 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:42.163694 21708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:42.163866 21708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:42.164899 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "df77bb88d4104ee79db3e0970cd37c10"
format_stamp: "Formatted at 2026-09-25 14:54:42 on dist-test-slave-gpc1"
server_key: "fb0b51419ba669a9ae56d4c1cf9b7f69"
server_key_iv: "f1d04288b6738cd0b3f94a187a8286f8"
server_key_version: "encryptionkey@0"
I20260925 14:54:42.165271 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "df77bb88d4104ee79db3e0970cd37c10"
format_stamp: "Formatted at 2026-09-25 14:54:42 on dist-test-slave-gpc1"
server_key: "fb0b51419ba669a9ae56d4c1cf9b7f69"
server_key_iv: "f1d04288b6738cd0b3f94a187a8286f8"
server_key_version: "encryptionkey@0"
I20260925 14:54:42.166640 21708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:54:42.167464 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:42.167659 21708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:42.167757 21708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "df77bb88d4104ee79db3e0970cd37c10"
format_stamp: "Formatted at 2026-09-25 14:54:42 on dist-test-slave-gpc1"
server_key: "fb0b51419ba669a9ae56d4c1cf9b7f69"
server_key_iv: "f1d04288b6738cd0b3f94a187a8286f8"
server_key_version: "encryptionkey@0"
I20260925 14:54:42.167838 21708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:42.181015 21708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:42.182886 21708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:42.183053 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:42.186836 21708 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35671
I20260925 14:54:42.186869 21775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35671 every 8 connection(s)
I20260925 14:54:42.187242 21708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:42.188207 21776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:42.190394 21776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10: Bootstrap starting.
I20260925 14:54:42.190982 21776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:42.191265 21776 log.cc:824] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:42.191941 21776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10: No bootstrap required, opened a new log
I20260925 14:54:42.193463 21776 raft_consensus.cc:348] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 14:54:42.193594 21776 raft_consensus.cc:374] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:42.193642 21776 raft_consensus.cc:729] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df77bb88d4104ee79db3e0970cd37c10, State: Initialized, Role: FOLLOWER
I20260925 14:54:42.193791 21776 consensus_queue.cc:260] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 14:54:42.193885 21776 raft_consensus.cc:388] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:42.193938 21776 raft_consensus.cc:482] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:42.193987 21776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:42.194518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21708
I20260925 14:54:42.194526 21776 raft_consensus.cc:504] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 14:54:42.194646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:42.194638 21776 leader_election.cc:302] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df77bb88d4104ee79db3e0970cd37c10; no voters:
I20260925 14:54:42.194833 392 external_mini_cluster.cc:1428] Setting key d1217b6bb18c4383847cfeebe5b15543
I20260925 14:54:42.194844 21776 leader_election.cc:288] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:42.194860 21779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:42.194976 21779 raft_consensus.cc:686] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [term 1 LEADER]: Becoming Leader. State: Replica: df77bb88d4104ee79db3e0970cd37c10, State: Running, Role: LEADER
I20260925 14:54:42.195092 21776 sys_catalog.cc:564] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:42.195137 21779 consensus_queue.cc:237] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } }
I20260925 14:54:42.195480 21779 sys_catalog.cc:455] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df77bb88d4104ee79db3e0970cd37c10. Latest consensus state: current_term: 1 leader_uuid: "df77bb88d4104ee79db3e0970cd37c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260925 14:54:42.195616 21779 sys_catalog.cc:458] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:42.195629 21780 sys_catalog.cc:455] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df77bb88d4104ee79db3e0970cd37c10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df77bb88d4104ee79db3e0970cd37c10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35671 } } }
I20260925 14:54:42.195811 21780 sys_catalog.cc:458] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:42.195900 21790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:42.196475 21790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:42.198045 21790 catalog_manager.cc:1347] Generated new cluster ID: d9a7100d51de4938beadfec25aacf6d0
I20260925 14:54:42.198096 21790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:42.200618 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:42.195274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:57415 (local address 127.0.0.1:35671)
0925 14:54:42.195628 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:42.195637 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:42.195669 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:54:42.195782 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:42.195788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:42.195848 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:42.196016 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:42.196259 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:42.197726 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:42.198700 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:42.199136 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:42.199259 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:42.199987 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:42.199991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:42.199991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:42.200164 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:42.200194 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:42.200205 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:42.200298 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:42.200377 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:54:42.200405 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":60,"threads_started":1}
I20260925 14:54:42.201493 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:42.206513 21790 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:42.207209 21790 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:42.209751 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:42.202787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:43687 (local address 127.0.0.1:35671)
0925 14:54:42.202823 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:42.202825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:42.206132 (+ 3307us) server_negotiation.cc:407] Connection header received
0925 14:54:42.206398 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:42.206399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:42.206412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:42.206477 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:42.206671 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:42.207366 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:42.208391 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:42.208703 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:42.208761 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:42.209498 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:42.209499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:42.209499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:42.209503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:42.209518 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:42.209523 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:42.209578 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:42.209668 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:54:42.209679 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":18}
I20260925 14:54:42.219565 21790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df77bb88d4104ee79db3e0970cd37c10: Generated new TSK 0
I20260925 14:54:42.219772 21790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:42.222033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21708
W20260925 14:54:42.226063 21693 connection.cc:535] client connection to 127.0.0.1:35671 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35671): Connection reset by peer (error 104)
2026-09-25T14:54:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:42 dist-test-slave-gpc1 krb5kdc[21807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:42 dist-test-slave-gpc1 krb5kdc[21807](info): set up 2 sockets
Sep 25 14:54:42 dist-test-slave-gpc1 krb5kdc[21807](info): commencing operation
krb5kdc: starting...
W20260925 14:54:43.266592 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:43 dist-test-slave-gpc1 krb5kdc[21807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:54:43.382427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43027
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:43.460683 21820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:43.460879 21820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:43.460902 21820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:43.462044 21820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:43.462093 21820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:43.462108 21820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:43.462121 21820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:43.462165 21820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:43.463629 21820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:43.463879 21820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:43.464102 21820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:43.468926 21825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:43.468922 21828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:43.468999 21820 server_base.cc:1034] running on GCE node
W20260925 14:54:43.468919 21826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:43.469340 21820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:43.469586 21820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:43.470723 21820 hybrid_clock.cc:648] HybridClock initialized: now 1790348083470710 us; error 29 us; skew 500 ppm
Sep 25 14:54:43 dist-test-slave-gpc1 krb5kdc[21807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:43.473122 21820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:54:43.473536 21820 webserver.cc:460] Webserver started at http://127.0.0.1:41369/ using document root <none> and password file <none>
I20260925 14:54:43.473749 21820 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:43.473794 21820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:43.473932 21820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:43.475015 21820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "55728fc06237493d80c355362bb7277b"
format_stamp: "Formatted at 2026-09-25 14:54:43 on dist-test-slave-gpc1"
server_key: "f54cd155944dedfed516862c825123e3"
server_key_iv: "f18b4754c044479fe2d6f0fd724ef3b6"
server_key_version: "encryptionkey@0"
I20260925 14:54:43.475359 21820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "55728fc06237493d80c355362bb7277b"
format_stamp: "Formatted at 2026-09-25 14:54:43 on dist-test-slave-gpc1"
server_key: "f54cd155944dedfed516862c825123e3"
server_key_iv: "f18b4754c044479fe2d6f0fd724ef3b6"
server_key_version: "encryptionkey@0"
I20260925 14:54:43.476706 21820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:43.477435 21835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:43.477605 21820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:43.477710 21820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "55728fc06237493d80c355362bb7277b"
format_stamp: "Formatted at 2026-09-25 14:54:43 on dist-test-slave-gpc1"
server_key: "f54cd155944dedfed516862c825123e3"
server_key_iv: "f18b4754c044479fe2d6f0fd724ef3b6"
server_key_version: "encryptionkey@0"
I20260925 14:54:43.477782 21820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:43.481052 21820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:43.482860 21820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:43.483017 21820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:43.486761 21820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43027
I20260925 14:54:43.486794 21887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43027 every 8 connection(s)
I20260925 14:54:43.487119 21820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:43.488046 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:43.490185 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b: Bootstrap starting.
I20260925 14:54:43.490744 21888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:43.490975 21888 log.cc:824] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:43.491585 21888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b: No bootstrap required, opened a new log
I20260925 14:54:43.493039 21888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } }
I20260925 14:54:43.493175 21888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:43.493196 21888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55728fc06237493d80c355362bb7277b, State: Initialized, Role: FOLLOWER
I20260925 14:54:43.493323 21888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } }
I20260925 14:54:43.493407 21888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:43.493429 21888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:43.493458 21888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:43.493925 21888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } }
I20260925 14:54:43.494007 21888 leader_election.cc:302] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55728fc06237493d80c355362bb7277b; no voters:
I20260925 14:54:43.494135 21888 leader_election.cc:288] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:43.494201 21891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:43.494392 21891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [term 1 LEADER]: Becoming Leader. State: Replica: 55728fc06237493d80c355362bb7277b, State: Running, Role: LEADER
I20260925 14:54:43.494433 21888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:43.494508 21891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } }
I20260925 14:54:43.494849 21892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55728fc06237493d80c355362bb7277b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } } }
I20260925 14:54:43.494858 21893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55728fc06237493d80c355362bb7277b. Latest consensus state: current_term: 1 leader_uuid: "55728fc06237493d80c355362bb7277b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55728fc06237493d80c355362bb7277b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43027 } } }
I20260925 14:54:43.494979 21892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:43.494995 21893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:43.495457 21895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:43.496027 21895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:43.496392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21820
I20260925 14:54:43.496552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:43.496752 392 external_mini_cluster.cc:1428] Setting key df66fb7fbe67c7d4ff3cac06a87b09c9
I20260925 14:54:43.497463 21895 catalog_manager.cc:1347] Generated new cluster ID: aa0bf5a5229b422791d96a858aaf4f3d
I20260925 14:54:43.497526 21895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:43.499259 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:43.497180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:44665 (local address 127.0.0.1:43027)
0925 14:54:43.497323 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:43.497336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:43.497662 (+ 326us) server_negotiation.cc:407] Connection header received
0925 14:54:43.497793 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:43.497801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:43.497829 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:43.497963 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:43.498828 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:43.498832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:43.498832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:43.498836 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:43.498874 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:43.498888 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:43.498948 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:43.499045 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:54:43.499093 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260925 14:54:43.500149 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:43.502718 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:43.501540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.184:56917 (local address 127.0.0.1:43027)
0925 14:54:43.501573 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:43.501575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:43.501778 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:54:43.501826 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:43.501827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:43.501844 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:43.501911 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:43.502498 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:43.502499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:43.502500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:43.502504 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:43.502519 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:43.502523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:43.502585 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:43.502644 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:54:43.502656 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:54:43.512586 21895 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:43.512759 21895 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:43.532780 21895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55728fc06237493d80c355362bb7277b: Generated new TSK 0
I20260925 14:54:43.532996 21895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:43.538722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21820
2026-09-25T14:54:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:43 dist-test-slave-gpc1 krb5kdc[21919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:43 dist-test-slave-gpc1 krb5kdc[21919](info): set up 2 sockets
Sep 25 14:54:43 dist-test-slave-gpc1 krb5kdc[21919](info): commencing operation
krb5kdc: starting...
W20260925 14:54:44.571694 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[21919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:44.684893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:44.765782 21932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:44.765987 21932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:44.766016 21932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:44.767242 21932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:44.767287 21932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:44.767300 21932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:44.767336 21932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:44.767365 21932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:44.768818 21932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:44.769060 21932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:44.769286 21932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:44.774022 21940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:44.774185 21932 server_base.cc:1034] running on GCE node
W20260925 14:54:44.774058 21938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:44.774024 21937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:44.774538 21932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:44.774837 21932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:44.776031 21932 hybrid_clock.cc:648] HybridClock initialized: now 1790348084776018 us; error 30 us; skew 500 ppm
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[21919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:44.778673 21932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:44.779103 21932 webserver.cc:460] Webserver started at http://127.0.98.62:44343/ using document root <none> and password file <none>
I20260925 14:54:44.779337 21932 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:44.779382 21932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:44.779546 21932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:44.780570 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "3ca9cfdffeca4ad2a524b80acf92b290"
format_stamp: "Formatted at 2026-09-25 14:54:44 on dist-test-slave-gpc1"
server_key: "8c185cd91d71f7f4d1cfcd871955eca1"
server_key_iv: "a7d7211130e15c0957fd98542bf904f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:44.780932 21932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ca9cfdffeca4ad2a524b80acf92b290"
format_stamp: "Formatted at 2026-09-25 14:54:44 on dist-test-slave-gpc1"
server_key: "8c185cd91d71f7f4d1cfcd871955eca1"
server_key_iv: "a7d7211130e15c0957fd98542bf904f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:44.782317 21932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:44.783084 21947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:44.783273 21932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:44.783370 21932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "3ca9cfdffeca4ad2a524b80acf92b290"
format_stamp: "Formatted at 2026-09-25 14:54:44 on dist-test-slave-gpc1"
server_key: "8c185cd91d71f7f4d1cfcd871955eca1"
server_key_iv: "a7d7211130e15c0957fd98542bf904f0"
server_key_version: "encryptionkey@0"
I20260925 14:54:44.783455 21932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:44.808292 21932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:44.810766 21932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:44.810935 21932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:44.814776 21932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35853
I20260925 14:54:44.814826 21999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35853 every 8 connection(s)
I20260925 14:54:44.815196 21932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:44.816118 22000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:44.818313 22000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290: Bootstrap starting.
I20260925 14:54:44.818892 22000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:44.819157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 21932
I20260925 14:54:44.819178 22000 log.cc:824] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:44.819285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:44.819480 392 external_mini_cluster.cc:1428] Setting key a63276f3375bdddefbe5e7ad337fc68b
I20260925 14:54:44.819932 22000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290: No bootstrap required, opened a new log
I20260925 14:54:44.821532 22000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } }
I20260925 14:54:44.821691 22000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:44.821776 22000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca9cfdffeca4ad2a524b80acf92b290, State: Initialized, Role: FOLLOWER
I20260925 14:54:44.821924 22000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } }
I20260925 14:54:44.822008 22000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:44.822031 22000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:44.822103 22000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:44.822947 22000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } }
I20260925 14:54:44.823065 22000 leader_election.cc:302] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ca9cfdffeca4ad2a524b80acf92b290; no voters:
I20260925 14:54:44.823247 22000 leader_election.cc:288] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:44.823280 22005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:44.823443 22005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [term 1 LEADER]: Becoming Leader. State: Replica: 3ca9cfdffeca4ad2a524b80acf92b290, State: Running, Role: LEADER
I20260925 14:54:44.823503 22000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:44.823632 22005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } }
I20260925 14:54:44.824231 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ca9cfdffeca4ad2a524b80acf92b290. Latest consensus state: current_term: 1 leader_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } } }
I20260925 14:54:44.824319 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:44.824513 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ca9cfdffeca4ad2a524b80acf92b290" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35853 } } }
I20260925 14:54:44.824648 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:44.824720 22014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:44.825253 22014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[21919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:44.827904 22014 catalog_manager.cc:1347] Generated new cluster ID: a8d43fedba09487793995a70696294b2
I20260925 14:54:44.827960 22014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:44.831987 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:44.819955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48267 (local address 127.0.98.62:35853)
0925 14:54:44.820121 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:44.820132 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:44.820165 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:54:44.821524 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:44.821531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:44.821582 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:44.821726 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:44.822515 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:44.823640 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:44.824847 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:44.825392 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:44.825516 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:44.828263 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:44.828266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:44.828267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:44.828277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:44.830101 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:44.830575 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:44.830578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:44.830578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:44.830702 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:44.830910 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:44.830910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:44.830911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:44.831176 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:44.831329 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:44.831696 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 14:54:44.831791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260925 14:54:44.833060 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:44.835158 22014 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:44.835880 22014 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:44.838855 22014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ca9cfdffeca4ad2a524b80acf92b290: Generated new TSK 0
I20260925 14:54:44.839021 22014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:44.842154 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:44.835009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35383 (local address 127.0.98.62:35853)
0925 14:54:44.835043 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:44.835045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:44.835191 (+ 146us) server_negotiation.cc:407] Connection header received
0925 14:54:44.835298 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:44.835299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:44.835308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:44.835336 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:44.835985 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:44.836718 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:44.837672 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:44.838002 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:44.838039 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:44.839557 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:44.839559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:44.839559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:44.839566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:44.840996 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:44.841367 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:44.841368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:44.841369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:44.841427 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:44.841648 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:44.841649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:44.841649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:44.841758 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:44.841867 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:44.841985 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:54:44.842049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:54:44.844235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 21932
W20260925 14:54:44.848687 21917 connection.cc:535] client connection to 127.0.98.62:35853 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35853): Connection reset by peer (error 104)
2026-09-25T14:54:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[22031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[22031](info): set up 2 sockets
Sep 25 14:54:44 dist-test-slave-gpc1 krb5kdc[22031](info): commencing operation
krb5kdc: starting...
W20260925 14:54:45.874523 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:45 dist-test-slave-gpc1 krb5kdc[22031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:45.987520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:46.068217 22044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:46.068419 22044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:46.068442 22044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:46.069591 22044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:46.069633 22044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:46.069646 22044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:46.069694 22044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:46.069713 22044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:46.071270 22044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:46.071557 22044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:46.071810 22044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:46.076674 22050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:46.076761 22044 server_base.cc:1034] running on GCE node
W20260925 14:54:46.076684 22052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:46.076686 22049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:46.077062 22044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:46.077294 22044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:46.078411 22044 hybrid_clock.cc:648] HybridClock initialized: now 1790348086078396 us; error 35 us; skew 500 ppm
Sep 25 14:54:46 dist-test-slave-gpc1 krb5kdc[22031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:46.089326 22044 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:46.089833 22044 webserver.cc:460] Webserver started at http://127.0.98.62:39637/ using document root <none> and password file <none>
I20260925 14:54:46.090087 22044 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:46.090158 22044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:46.090307 22044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:46.091447 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "1e622aed969848aba8314476f2241901"
format_stamp: "Formatted at 2026-09-25 14:54:46 on dist-test-slave-gpc1"
server_key: "43e94b713b35de2bffdfd4fa1d26b203"
server_key_iv: "e39c619d191739bd1266ff06d1a36f0b"
server_key_version: "encryptionkey@0"
I20260925 14:54:46.091861 22044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e622aed969848aba8314476f2241901"
format_stamp: "Formatted at 2026-09-25 14:54:46 on dist-test-slave-gpc1"
server_key: "43e94b713b35de2bffdfd4fa1d26b203"
server_key_iv: "e39c619d191739bd1266ff06d1a36f0b"
server_key_version: "encryptionkey@0"
I20260925 14:54:46.093174 22044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:46.093932 22059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:46.094125 22044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:46.094224 22044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "1e622aed969848aba8314476f2241901"
format_stamp: "Formatted at 2026-09-25 14:54:46 on dist-test-slave-gpc1"
server_key: "43e94b713b35de2bffdfd4fa1d26b203"
server_key_iv: "e39c619d191739bd1266ff06d1a36f0b"
server_key_version: "encryptionkey@0"
I20260925 14:54:46.094305 22044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:46.110895 22044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:46.113247 22044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:46.113432 22044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:46.118081 22044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39849
I20260925 14:54:46.118134 22111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39849 every 8 connection(s)
I20260925 14:54:46.118645 22044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:46.119529 22112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:46.121739 22112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: Bootstrap starting.
I20260925 14:54:46.122313 22112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:46.122315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22044
I20260925 14:54:46.122471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:46.122682 22112 log.cc:824] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:46.122697 392 external_mini_cluster.cc:1428] Setting key 69c3615b111ff401d5f5fed0370c9829
I20260925 14:54:46.123617 22112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: No bootstrap required, opened a new log
I20260925 14:54:46.125769 22112 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } }
I20260925 14:54:46.125936 22112 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:46.125967 22112 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e622aed969848aba8314476f2241901, State: Initialized, Role: FOLLOWER
I20260925 14:54:46.126073 22112 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } }
I20260925 14:54:46.126142 22112 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:46.126170 22112 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:46.126206 22112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:46.126776 22112 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } }
I20260925 14:54:46.126904 22112 leader_election.cc:302] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e622aed969848aba8314476f2241901; no voters:
I20260925 14:54:46.127075 22112 leader_election.cc:288] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:46.127099 22117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:46.127305 22117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [term 1 LEADER]: Becoming Leader. State: Replica: 1e622aed969848aba8314476f2241901, State: Running, Role: LEADER
I20260925 14:54:46.127347 22112 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:46.127447 22117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } }
I20260925 14:54:46.127885 22118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e622aed969848aba8314476f2241901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } }
I20260925 14:54:46.127866 22117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e622aed969848aba8314476f2241901. Latest consensus state: current_term: 1 leader_uuid: "1e622aed969848aba8314476f2241901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e622aed969848aba8314476f2241901" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39849 } } }
I20260925 14:54:46.128026 22118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:46.128034 22117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:46.128453 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:46.123193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38099 (local address 127.0.98.62:39849)
0925 14:54:46.123359 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:46.123367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:46.123394 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:46.123883 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:46.123890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:46.123937 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:46.124051 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:46.124282 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:46.125149 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:46.126297 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:46.126768 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:46.126853 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:46.127682 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:46.127686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:46.127686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:46.127922 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:46.127962 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:46.127977 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:46.128034 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:46.128196 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:54:46.128227 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
W20260925 14:54:46.129463 22132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:46.129515 22132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:46.129570 22133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:46.129722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:46.129767 22133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:46.131397 22133 catalog_manager.cc:1347] Generated new cluster ID: 79b1b936046f4e1a9940be39be361b65
I20260925 14:54:46.131453 22133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:46.134011 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:46.131125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35975 (local address 127.0.98.62:39849)
0925 14:54:46.131158 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:46.131159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:46.131322 (+ 163us) server_negotiation.cc:407] Connection header received
0925 14:54:46.131370 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:46.131371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:46.131381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:46.131408 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:46.131623 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:46.132089 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:46.132811 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:46.133039 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:46.133114 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:46.133775 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:46.133777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:46.133777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:46.133781 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:46.133795 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:46.133799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:46.133852 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:46.133910 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:54:46.133918 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:46.141197 22133 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:46.141688 22133 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:46.151443 22133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e622aed969848aba8314476f2241901: Generated new TSK 0
I20260925 14:54:46.151695 22133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:46.155627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22044
W20260925 14:54:46.159754 22029 connection.cc:535] client connection to 127.0.98.62:39849 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39849): Connection reset by peer (error 104)
2026-09-25T14:54:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:46 dist-test-slave-gpc1 krb5kdc[22143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:46 dist-test-slave-gpc1 krb5kdc[22143](info): set up 2 sockets
Sep 25 14:54:46 dist-test-slave-gpc1 krb5kdc[22143](info): commencing operation
krb5kdc: starting...
W20260925 14:54:47.194514 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:47 dist-test-slave-gpc1 krb5kdc[22143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:47.311110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:47.390048 22156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:47.390275 22156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:47.390313 22156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:47.391461 22156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:47.391535 22156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:47.391561 22156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:47.391589 22156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:47.391610 22156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:47.393054 22156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:47.393332 22156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:47.393548 22156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:47.398237 22164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:47.398257 22161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:47.398265 22156 server_base.cc:1034] running on GCE node
W20260925 14:54:47.398231 22162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:47.398661 22156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:47.399027 22156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:47.400204 22156 hybrid_clock.cc:648] HybridClock initialized: now 1790348087400184 us; error 30 us; skew 500 ppm
Sep 25 14:54:47 dist-test-slave-gpc1 krb5kdc[22143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:47.402616 22156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:47.403119 22156 webserver.cc:460] Webserver started at http://127.0.98.62:38459/ using document root <none> and password file <none>
I20260925 14:54:47.403364 22156 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:47.403431 22156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:47.403573 22156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:47.404594 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "76478375e7134579ace951541c592ea6"
format_stamp: "Formatted at 2026-09-25 14:54:47 on dist-test-slave-gpc1"
server_key: "177dd2d4862fe3bcc78803c187db41dc"
server_key_iv: "b1a2af4ab319cd27349d87b09260ed4e"
server_key_version: "encryptionkey@0"
I20260925 14:54:47.404959 22156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "76478375e7134579ace951541c592ea6"
format_stamp: "Formatted at 2026-09-25 14:54:47 on dist-test-slave-gpc1"
server_key: "177dd2d4862fe3bcc78803c187db41dc"
server_key_iv: "b1a2af4ab319cd27349d87b09260ed4e"
server_key_version: "encryptionkey@0"
I20260925 14:54:47.406237 22156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:47.406986 22171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:47.407150 22156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:47.407248 22156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "76478375e7134579ace951541c592ea6"
format_stamp: "Formatted at 2026-09-25 14:54:47 on dist-test-slave-gpc1"
server_key: "177dd2d4862fe3bcc78803c187db41dc"
server_key_iv: "b1a2af4ab319cd27349d87b09260ed4e"
server_key_version: "encryptionkey@0"
I20260925 14:54:47.407328 22156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:47.410615 22156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:47.412351 22156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:47.412508 22156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:47.416514 22156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43149
I20260925 14:54:47.416544 22223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43149 every 8 connection(s)
I20260925 14:54:47.416864 22156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:47.417761 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:47.420009 22224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: Bootstrap starting.
I20260925 14:54:47.420552 22224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:47.420825 22224 log.cc:824] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:47.421517 22224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: No bootstrap required, opened a new log
I20260925 14:54:47.423137 22224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260925 14:54:47.423269 22224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:47.423290 22224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76478375e7134579ace951541c592ea6, State: Initialized, Role: FOLLOWER
I20260925 14:54:47.423465 22224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260925 14:54:47.423530 22224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:47.423552 22224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:47.423617 22224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:47.424139 22224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260925 14:54:47.424219 22224 leader_election.cc:302] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76478375e7134579ace951541c592ea6; no voters:
I20260925 14:54:47.424405 22224 leader_election.cc:288] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:47.424434 22227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:47.424640 22227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [term 1 LEADER]: Becoming Leader. State: Replica: 76478375e7134579ace951541c592ea6, State: Running, Role: LEADER
I20260925 14:54:47.424651 22224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:47.424801 22227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } }
I20260925 14:54:47.425194 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76478375e7134579ace951541c592ea6. Latest consensus state: current_term: 1 leader_uuid: "76478375e7134579ace951541c592ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } }
I20260925 14:54:47.425535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22156
I20260925 14:54:47.425647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:47.425848 392 external_mini_cluster.cc:1428] Setting key 3d57f8feac05c996eda229ebadf16bf6
I20260925 14:54:47.425309 22228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76478375e7134579ace951541c592ea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76478375e7134579ace951541c592ea6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43149 } } }
I20260925 14:54:47.426198 22228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:47.426405 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6 [sys.catalog]: This master's current role is: LEADER
W20260925 14:54:47.426606 22241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:47.426743 22241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:47.427021 22243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:47.427213 22243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:47.428634 22243 catalog_manager.cc:1347] Generated new cluster ID: 920cd81ff6d3409fae46436680dfaf5c
I20260925 14:54:47.428691 22243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:47.439395 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:47.426555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51025 (local address 127.0.98.62:43149)
0925 14:54:47.426710 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:47.426719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:47.426751 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:54:47.426866 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:47.426873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:47.426905 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:47.427077 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:47.438909 (+ 11832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:47.438914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:47.438915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:47.438921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:47.438975 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:47.438993 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:47.439070 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:47.439138 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:54:47.439184 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260925 14:54:47.440410 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:47.442982 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:47.441945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55585 (local address 127.0.98.62:43149)
0925 14:54:47.441981 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:47.441983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:47.442090 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:54:47.442119 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:47.442120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:47.442125 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:47.442191 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:47.442767 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:47.442769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:47.442769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:47.442774 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:47.442790 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:47.442794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:47.442856 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:47.442906 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:54:47.442918 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:47.452605 22243 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:47.452791 22243 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:47.455747 22243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76478375e7134579ace951541c592ea6: Generated new TSK 0
I20260925 14:54:47.455938 22243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:47.459884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22156
2026-09-25T14:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:47 dist-test-slave-gpc1 krb5kdc[22255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:47 dist-test-slave-gpc1 krb5kdc[22255](info): set up 2 sockets
Sep 25 14:54:47 dist-test-slave-gpc1 krb5kdc[22255](info): commencing operation
krb5kdc: starting...
W20260925 14:54:48.562395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:48.677665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:48.765307 22268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:48.765507 22268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:48.765532 22268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:48.766669 22268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:48.766709 22268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:48.766723 22268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:48.766736 22268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:48.766748 22268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:48.768167 22268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:48.768344 22268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:48.768501 22268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:48.773447 22273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:48.773471 22274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:48.773496 22268 server_base.cc:1034] running on GCE node
W20260925 14:54:48.773447 22276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:48.773943 22268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:48.774261 22268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:48.775434 22268 hybrid_clock.cc:648] HybridClock initialized: now 1790348088775405 us; error 50 us; skew 500 ppm
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:48.777940 22268 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:48.778313 22268 webserver.cc:460] Webserver started at http://127.0.98.62:43229/ using document root <none> and password file <none>
I20260925 14:54:48.778573 22268 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:48.778617 22268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:48.778704 22268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:48.779677 22268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "ee6972e15d4b4c7e9d474a202c0746cf"
format_stamp: "Formatted at 2026-09-25 14:54:48 on dist-test-slave-gpc1"
server_key: "764953d6de61361fcb8f8c53a1c41d50"
server_key_iv: "e26b143d3e677dfac75a42e4f623c95c"
server_key_version: "encryptionkey@0"
I20260925 14:54:48.779968 22268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee6972e15d4b4c7e9d474a202c0746cf"
format_stamp: "Formatted at 2026-09-25 14:54:48 on dist-test-slave-gpc1"
server_key: "764953d6de61361fcb8f8c53a1c41d50"
server_key_iv: "e26b143d3e677dfac75a42e4f623c95c"
server_key_version: "encryptionkey@0"
I20260925 14:54:48.781162 22268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:54:48.781869 22283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:48.782047 22268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:48.782122 22268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "ee6972e15d4b4c7e9d474a202c0746cf"
format_stamp: "Formatted at 2026-09-25 14:54:48 on dist-test-slave-gpc1"
server_key: "764953d6de61361fcb8f8c53a1c41d50"
server_key_iv: "e26b143d3e677dfac75a42e4f623c95c"
server_key_version: "encryptionkey@0"
I20260925 14:54:48.782176 22268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:48.799301 22268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:48.801091 22268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:48.801199 22268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:48.805054 22268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41723
I20260925 14:54:48.805104 22335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41723 every 8 connection(s)
I20260925 14:54:48.805425 22268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:48.806317 22336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:48.808593 22336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf: Bootstrap starting.
I20260925 14:54:48.809119 22336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:48.809346 22336 log.cc:824] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:48.810043 22336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf: No bootstrap required, opened a new log
I20260925 14:54:48.811654 22336 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260925 14:54:48.811790 22336 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:48.811812 22336 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6972e15d4b4c7e9d474a202c0746cf, State: Initialized, Role: FOLLOWER
I20260925 14:54:48.811913 22336 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260925 14:54:48.811993 22336 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:48.812016 22336 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:48.812043 22336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:48.812247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22268
I20260925 14:54:48.812361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:48.812558 392 external_mini_cluster.cc:1428] Setting key 5c6379fcf44b1c35e1a5a6798bee377a
I20260925 14:54:48.812680 22336 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260925 14:54:48.812803 22336 leader_election.cc:302] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee6972e15d4b4c7e9d474a202c0746cf; no voters:
I20260925 14:54:48.812968 22336 leader_election.cc:288] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:48.813009 22339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:48.813199 22339 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [term 1 LEADER]: Becoming Leader. State: Replica: ee6972e15d4b4c7e9d474a202c0746cf, State: Running, Role: LEADER
I20260925 14:54:48.813313 22336 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:48.813313 22339 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } }
I20260925 14:54:48.813731 22339 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee6972e15d4b4c7e9d474a202c0746cf. Latest consensus state: current_term: 1 leader_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } } }
I20260925 14:54:48.813773 22343 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6972e15d4b4c7e9d474a202c0746cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41723 } } }
I20260925 14:54:48.813840 22339 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:48.813861 22343 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:48.814447 22345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:48.815142 22345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:48.817615 22345 catalog_manager.cc:1347] Generated new cluster ID: ce2be74a02364528acaa121dad1cba37
I20260925 14:54:48.817669 22345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:48.825949 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:48.813051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52417 (local address 127.0.98.62:41723)
0925 14:54:48.813219 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:48.813229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:48.815302 (+ 2073us) server_negotiation.cc:407] Connection header received
0925 14:54:48.815530 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:48.815538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:48.815594 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:48.815735 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:48.816446 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:48.817463 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:48.818527 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:48.819190 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:48.819306 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:48.822278 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:48.822282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:48.822282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:48.822295 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:48.824262 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:48.824681 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:48.824684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:48.824684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:48.824795 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:48.825047 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:48.825047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:48.825048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:48.825337 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:48.825494 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:48.825601 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:54:48.825733 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260925 14:54:48.827152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:48.832667 22345 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:48.833288 22345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:48.837384 22341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:48.828596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40471 (local address 127.0.98.62:41723)
0925 14:54:48.828627 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:48.828629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:48.830516 (+ 1887us) server_negotiation.cc:407] Connection header received
0925 14:54:48.830577 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:48.830578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:48.830587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:48.830639 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:48.831227 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:48.831958 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:48.832978 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:48.833309 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:48.833382 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:48.834777 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:48.834778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:48.834778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:48.834785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:48.836285 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:48.836670 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:48.836671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:48.836672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:48.836722 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:48.836913 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:48.836913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:48.836914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:48.837021 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:48.837113 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:48.837227 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:54:48.837303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 14:54:48.838840 22345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee6972e15d4b4c7e9d474a202c0746cf: Generated new TSK 0
I20260925 14:54:48.838995 22345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:48.841886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22268
2026-09-25T14:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22367](info): set up 2 sockets
Sep 25 14:54:48 dist-test-slave-gpc1 krb5kdc[22367](info): commencing operation
krb5kdc: starting...
W20260925 14:54:49.870543 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:49 dist-test-slave-gpc1 krb5kdc[22367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:49.984365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:50.064200 22380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:50.064428 22380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:50.064465 22380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:50.065622 22380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:50.065687 22380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:50.065711 22380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:50.065732 22380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:50.065761 22380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:50.067289 22380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:50.067562 22380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:50.067799 22380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:50.072636 22388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:50.072615 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:50.072646 22380 server_base.cc:1034] running on GCE node
W20260925 14:54:50.072611 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:50.073073 22380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:50.073390 22380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:50.074548 22380 hybrid_clock.cc:648] HybridClock initialized: now 1790348090074537 us; error 39 us; skew 500 ppm
Sep 25 14:54:50 dist-test-slave-gpc1 krb5kdc[22367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:50.076926 22380 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:50.077346 22380 webserver.cc:460] Webserver started at http://127.0.98.62:35089/ using document root <none> and password file <none>
I20260925 14:54:50.077572 22380 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:50.077641 22380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:50.077795 22380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:50.078930 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "0bc2ccf6593e4615bb6d910680eb871e"
format_stamp: "Formatted at 2026-09-25 14:54:50 on dist-test-slave-gpc1"
server_key: "04a9b6d6e0afd2efc1c010d0ad59b48a"
server_key_iv: "edbd5555f27841d4645b17f3e03032cb"
server_key_version: "encryptionkey@0"
I20260925 14:54:50.079288 22380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "0bc2ccf6593e4615bb6d910680eb871e"
format_stamp: "Formatted at 2026-09-25 14:54:50 on dist-test-slave-gpc1"
server_key: "04a9b6d6e0afd2efc1c010d0ad59b48a"
server_key_iv: "edbd5555f27841d4645b17f3e03032cb"
server_key_version: "encryptionkey@0"
I20260925 14:54:50.080592 22380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:50.081310 22395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:50.081496 22380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:50.081598 22380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "0bc2ccf6593e4615bb6d910680eb871e"
format_stamp: "Formatted at 2026-09-25 14:54:50 on dist-test-slave-gpc1"
server_key: "04a9b6d6e0afd2efc1c010d0ad59b48a"
server_key_iv: "edbd5555f27841d4645b17f3e03032cb"
server_key_version: "encryptionkey@0"
I20260925 14:54:50.081670 22380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:50.114746 22380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:50.116703 22380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:50.116889 22380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:50.121039 22380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37177
I20260925 14:54:50.121073 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37177 every 8 connection(s)
I20260925 14:54:50.121466 22380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:50.122328 22448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:50.124632 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e: Bootstrap starting.
I20260925 14:54:50.125146 22448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:50.125365 22448 log.cc:824] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:50.126021 22448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e: No bootstrap required, opened a new log
I20260925 14:54:50.127506 22448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260925 14:54:50.127652 22448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:50.127676 22448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc2ccf6593e4615bb6d910680eb871e, State: Initialized, Role: FOLLOWER
I20260925 14:54:50.127830 22448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260925 14:54:50.127912 22448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:50.127934 22448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:50.127962 22448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:50.128412 22448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260925 14:54:50.128491 22448 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc2ccf6593e4615bb6d910680eb871e; no voters:
I20260925 14:54:50.128614 22448 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:50.128667 22451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:50.128827 22451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [term 1 LEADER]: Becoming Leader. State: Replica: 0bc2ccf6593e4615bb6d910680eb871e, State: Running, Role: LEADER
I20260925 14:54:50.128865 22448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:50.128922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22380
I20260925 14:54:50.128918 22451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } }
I20260925 14:54:50.129045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:50.129199 22451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc2ccf6593e4615bb6d910680eb871e. Latest consensus state: current_term: 1 leader_uuid: "0bc2ccf6593e4615bb6d910680eb871e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } } }
I20260925 14:54:50.129249 392 external_mini_cluster.cc:1428] Setting key 2e839cfcca85f8c5ebea3afa87739ea0
I20260925 14:54:50.129281 22451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:50.129439 22453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc2ccf6593e4615bb6d910680eb871e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc2ccf6593e4615bb6d910680eb871e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37177 } } }
I20260925 14:54:50.129559 22453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:50.130362 22457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:50.131182 22457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:50.132691 22457 catalog_manager.cc:1347] Generated new cluster ID: 389a3b20b016452b89ff32ebdc5d4f77
I20260925 14:54:50.132740 22457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:50.135108 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:50.129705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51783 (local address 127.0.98.62:37177)
0925 14:54:50.129875 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:50.129884 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:50.129915 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:54:50.130056 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:50.130063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:50.130120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:50.130275 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:50.130607 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:50.131916 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:50.133012 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:50.133608 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:50.133722 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:50.134423 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:50.134426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:50.134427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:50.134646 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:50.134689 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:50.134703 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:50.134776 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:50.134841 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:54:50.134907 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260925 14:54:50.136013 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:50.141115 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:50.137421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58281 (local address 127.0.98.62:37177)
0925 14:54:50.137455 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:50.137456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:50.137614 (+ 158us) server_negotiation.cc:407] Connection header received
0925 14:54:50.137643 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:50.137643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:50.137655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:50.137720 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:50.137837 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:50.138912 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:50.139849 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:50.140181 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:50.140265 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:50.140894 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:50.140895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:50.140896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:50.140901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:50.140914 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:50.140918 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:50.140973 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:50.141030 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:54:50.141053 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:50.141709 22457 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:50.142118 22457 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:50.151055 22457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc2ccf6593e4615bb6d910680eb871e: Generated new TSK 0
I20260925 14:54:50.151237 22457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:50.154827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22380
2026-09-25T14:54:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:50 dist-test-slave-gpc1 krb5kdc[22479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:50 dist-test-slave-gpc1 krb5kdc[22479](info): set up 2 sockets
Sep 25 14:54:50 dist-test-slave-gpc1 krb5kdc[22479](info): commencing operation
krb5kdc: starting...
W20260925 14:54:51.182581 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:51 dist-test-slave-gpc1 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:51.292807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:51.370113 22492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:51.370353 22492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:51.370428 22492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:51.371667 22492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:51.371744 22492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:51.371774 22492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:51.371801 22492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:51.371827 22492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:51.373314 22492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:51.373588 22492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:51.373808 22492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:51.378629 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:51.378659 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:51.378870 22492 server_base.cc:1034] running on GCE node
W20260925 14:54:51.378880 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:51.379156 22492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:51.379443 22492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:51.380607 22492 hybrid_clock.cc:648] HybridClock initialized: now 1790348091380598 us; error 28 us; skew 500 ppm
Sep 25 14:54:51 dist-test-slave-gpc1 krb5kdc[22479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:51.388934 22492 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:51.389333 22492 webserver.cc:460] Webserver started at http://127.0.98.62:36709/ using document root <none> and password file <none>
I20260925 14:54:51.389524 22492 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:51.389564 22492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:51.389652 22492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:51.390722 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "1e75ffafe89a4b48abea540e11dae5ec"
format_stamp: "Formatted at 2026-09-25 14:54:51 on dist-test-slave-gpc1"
server_key: "91783495f809df67b766da6db761fbdf"
server_key_iv: "b1b8f9c6ab600a064c25e4eeb9c62fea"
server_key_version: "encryptionkey@0"
I20260925 14:54:51.391027 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e75ffafe89a4b48abea540e11dae5ec"
format_stamp: "Formatted at 2026-09-25 14:54:51 on dist-test-slave-gpc1"
server_key: "91783495f809df67b766da6db761fbdf"
server_key_iv: "b1b8f9c6ab600a064c25e4eeb9c62fea"
server_key_version: "encryptionkey@0"
I20260925 14:54:51.392277 22492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:51.392963 22507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:51.393157 22492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:51.393244 22492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "1e75ffafe89a4b48abea540e11dae5ec"
format_stamp: "Formatted at 2026-09-25 14:54:51 on dist-test-slave-gpc1"
server_key: "91783495f809df67b766da6db761fbdf"
server_key_iv: "b1b8f9c6ab600a064c25e4eeb9c62fea"
server_key_version: "encryptionkey@0"
I20260925 14:54:51.393299 22492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:51.396404 22492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:51.398537 22492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:51.398638 22492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:51.402194 22492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35127
I20260925 14:54:51.402256 22559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35127 every 8 connection(s)
I20260925 14:54:51.402585 22492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:51.403421 22560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:51.405666 22560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec: Bootstrap starting.
I20260925 14:54:51.406227 22560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:51.406564 22560 log.cc:824] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:51.407070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22492
I20260925 14:54:51.407189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:51.407307 22560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec: No bootstrap required, opened a new log
I20260925 14:54:51.407392 392 external_mini_cluster.cc:1428] Setting key bb521ebfd223f54d9d4cf0479d4bd1f5
I20260925 14:54:51.409703 22560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } }
I20260925 14:54:51.409840 22560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:54:51.409863 22560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e75ffafe89a4b48abea540e11dae5ec, State: Initialized, Role: FOLLOWER
I20260925 14:54:51.409997 22560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } }
I20260925 14:54:51.410045 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:51.407880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54425 (local address 127.0.98.62:35127)
0925 14:54:51.408066 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:51.408074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:51.408097 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:54:51.408628 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:51.408634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:51.408657 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:51.408768 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:51.409360 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:51.409363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:51.409363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:51.409367 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:51.409404 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:51.409413 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:51.409465 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:51.409544 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:54:51.409615 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 14:54:51.410077 22560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:54:51.410100 22560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:54:51.410156 22560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:51.410868 22560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } }
I20260925 14:54:51.410992 22560 leader_election.cc:302] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e75ffafe89a4b48abea540e11dae5ec; no voters:
I20260925 14:54:51.411124 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:51.411156 22560 leader_election.cc:288] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:54:51.411190 22565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:51.411350 22565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [term 1 LEADER]: Becoming Leader. State: Replica: 1e75ffafe89a4b48abea540e11dae5ec, State: Running, Role: LEADER
I20260925 14:54:51.411369 22560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:51.412694 22565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } }
I20260925 14:54:51.413105 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e75ffafe89a4b48abea540e11dae5ec. Latest consensus state: current_term: 1 leader_uuid: "1e75ffafe89a4b48abea540e11dae5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } } }
I20260925 14:54:51.413237 22566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e75ffafe89a4b48abea540e11dae5ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e75ffafe89a4b48abea540e11dae5ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35127 } } }
I20260925 14:54:51.413304 22566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:51.413264 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:51.413513 22584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:51.414089 22584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:51.414170 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:51.413046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39461 (local address 127.0.98.62:35127)
0925 14:54:51.413097 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:51.413099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:51.413108 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:51.413127 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:51.413128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:51.413132 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:51.413188 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:51.413790 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:51.413791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:51.413792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:54:51.413796 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:51.413810 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:54:51.413814 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:51.413866 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:51.414055 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:54:51.414066 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:54:51.415575 22584 catalog_manager.cc:1347] Generated new cluster ID: c761b65bb34549f2a065a43a60d09a23
I20260925 14:54:51.415632 22584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:51.421445 22584 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:54:51.421593 22584 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:51.425861 22584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e75ffafe89a4b48abea540e11dae5ec: Generated new TSK 0
I20260925 14:54:51.426026 22584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:51.431289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22492
2026-09-25T14:54:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1276 ms)
[----------] 18 tests from AuthTokenIssuingTest (23657 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:51 dist-test-slave-gpc1 krb5kdc[22591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:51 dist-test-slave-gpc1 krb5kdc[22591](info): set up 2 sockets
Sep 25 14:54:51 dist-test-slave-gpc1 krb5kdc[22591](info): commencing operation
krb5kdc: starting...
W20260925 14:54:52.462584 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:52.587286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:52.668512 22604 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:52.668725 22604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:52.668743 22604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:52.668762 22604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:52.669836 22604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:52.669870 22604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:52.669884 22604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:52.669898 22604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:52.669909 22604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:52.671335 22604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:52.671646 22604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:52.671881 22604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:52.676724 22610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:52.676746 22609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:52.676837 22604 server_base.cc:1034] running on GCE node
W20260925 14:54:52.676731 22612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:52.677199 22604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:52.677513 22604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:52.678655 22604 hybrid_clock.cc:648] HybridClock initialized: now 1790348092678634 us; error 35 us; skew 500 ppm
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:52.680968 22604 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:52.681427 22604 webserver.cc:460] Webserver started at http://127.0.98.62:42611/ using document root <none> and password file <none>
I20260925 14:54:52.681655 22604 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:52.681725 22604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:52.681878 22604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:52.682996 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "2e63f67b78ce4e37913b42475bca1407"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "ff36229427e43d419767a9080b304127"
server_key_iv: "a6dbb8d919cc586442efb5285ba9fbb4"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.683360 22604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e63f67b78ce4e37913b42475bca1407"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "ff36229427e43d419767a9080b304127"
server_key_iv: "a6dbb8d919cc586442efb5285ba9fbb4"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.684994 22604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:52.685717 22619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:52.685879 22604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:52.685987 22604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "2e63f67b78ce4e37913b42475bca1407"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "ff36229427e43d419767a9080b304127"
server_key_iv: "a6dbb8d919cc586442efb5285ba9fbb4"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.686074 22604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:52.698700 22604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:52.700876 22604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:52.701047 22604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:52.704942 22604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45965
I20260925 14:54:52.704988 22671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45965 every 8 connection(s)
I20260925 14:54:52.705370 22604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:52.706423 22672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:52.707942 22672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:52.712072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22604
I20260925 14:54:52.712182 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:52.712369 392 external_mini_cluster.cc:1428] Setting key d51c08be0dce176bbd4d8322211a6b0d
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:52.718722 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.708660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.0.98.62:45965)
0925 14:54:52.708819 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:52.708830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:52.708970 (+ 140us) server_negotiation.cc:407] Connection header received
0925 14:54:52.709546 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:52.709548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:52.709569 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:52.709661 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:52.710215 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.711096 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.711989 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.712382 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.712504 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.715329 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:52.715337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:52.715337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:52.715343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:52.717081 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.717566 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.717569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.717569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.717643 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.717915 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.717915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.717915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.718123 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:52.718182 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:52.718451 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:54:52.718566 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260925 14:54:52.718811 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.708595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45965 (local address 127.0.0.1:37102)
0925 14:54:52.708824 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.708868 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:52.709372 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:52.709669 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:52.709671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:52.709963 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:52.710107 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.710112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.711201 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.711202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.711890 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.711892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.712537 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.712538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.712634 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.713141 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:52.713151 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:52.715174 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:52.717169 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.717175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.717178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.717484 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.717713 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.717714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.717715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.717832 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.718192 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:52.718200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:52.718283 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 14:54:52.718427 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 14:54:52.718654 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":70,"thread_start_us":89,"threads_started":1}
I20260925 14:54:52.719718 22672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:52.720104 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.719972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55500)
0925 14:54:52.720027 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.720077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260925 14:54:52.720261 22622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
I20260925 14:54:52.720695 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.720589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55508)
0925 14:54:52.720650 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.720677 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 14:54:52.720830 22672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:54:52.736400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42341
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:52.770077 22672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } attempt: 1
I20260925 14:54:52.770640 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.770489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55512)
0925 14:54:52.770555 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.770606 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:54:52.771020 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.770926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55518)
0925 14:54:52.770973 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.771000 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:54:52.771160 22672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
W20260925 14:54:52.817327 22679 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:52.817546 22679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:52.817564 22679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:52.817610 22679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:52.818823 22679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:52.818867 22679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:52.818882 22679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:52.818915 22679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:52.818941 22679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:52.820513 22679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:52.820775 22679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:52.821014 22679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:52.825893 22684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:52.825905 22685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:52.826064 22679 server_base.cc:1034] running on GCE node
W20260925 14:54:52.826195 22687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:52.826498 22679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:52.826807 22679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:52.827980 22679 hybrid_clock.cc:648] HybridClock initialized: now 1790348092827974 us; error 36 us; skew 500 ppm
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:52.830320 22679 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:54:52.830802 22679 webserver.cc:460] Webserver started at http://127.0.98.61:40951/ using document root <none> and password file <none>
I20260925 14:54:52.831025 22679 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:52.831091 22679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:52.831230 22679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:52.832264 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data/instance:
uuid: "ed0c2c081c5d439c8c7cc5750e3986fc"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "cccba067c9a47574d16b0e679504be2b"
server_key_iv: "0d6178de9b65d2a89ccb7a74629a9b63"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.832634 22679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal/instance:
uuid: "ed0c2c081c5d439c8c7cc5750e3986fc"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "cccba067c9a47574d16b0e679504be2b"
server_key_iv: "0d6178de9b65d2a89ccb7a74629a9b63"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.833901 22679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:52.834646 22694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:52.834810 22679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:54:52.834904 22679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "ed0c2c081c5d439c8c7cc5750e3986fc"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "cccba067c9a47574d16b0e679504be2b"
server_key_iv: "0d6178de9b65d2a89ccb7a74629a9b63"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.834986 22679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:52.839331 22672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } attempt: 2
I20260925 14:54:52.839733 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.839607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55534)
0925 14:54:52.839667 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.839703 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:54:52.840080 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.839987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55540)
0925 14:54:52.840031 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.840057 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:54:52.840186 22672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42341: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42341: connect: Connection refused (error 111)
I20260925 14:54:52.852710 22679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:52.854574 22679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:52.854727 22679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:52.858626 22679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42341
I20260925 14:54:52.858793 22746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42341 every 8 connection(s)
I20260925 14:54:52.859026 22679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:54:52.859994 22747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:52.860863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22679
I20260925 14:54:52.860987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-1/wal/instance
I20260925 14:54:52.861164 392 external_mini_cluster.cc:1428] Setting key e6e18a4de38e5f5efb41244dbf2e9401
I20260925 14:54:52.861660 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260925 14:54:52.870582 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.862325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.0.98.62:45965)
0925 14:54:52.862402 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:52.862404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:52.862627 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:54:52.863113 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:52.863114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:52.863130 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:52.863192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:52.863782 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.864261 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.865236 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.865456 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.865542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.867641 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:52.867642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:52.867642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:52.867651 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:52.869099 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.869557 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.869558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.869558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.869618 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.869893 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.869894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.869895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.870029 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:52.870092 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:52.870436 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 14:54:52.870518 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:54:52.870867 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.862329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45965 (local address 127.0.0.1:37114)
0925 14:54:52.862493 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.862531 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:52.863011 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:52.863265 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:52.863268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:52.863503 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:52.863683 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.863687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.864371 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.864373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.865144 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.865146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.865571 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.865572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.865683 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.866206 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:52.866214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:52.867550 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:52.869197 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.869204 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.869206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.869443 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.869686 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.869687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.869687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.869792 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.870125 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:52.870130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:52.870240 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:54:52.870414 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 14:54:52.870588 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260925 14:54:52.871493 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:52.879382 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.871773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55546)
0925 14:54:52.871839 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.871850 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:52.871894 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:52.872172 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:52.872172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:52.872277 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:52.872318 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.872320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.873614 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.873615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.874093 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.874095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.874823 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.874824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.874878 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.875262 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:52.875266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:52.876614 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:52.878469 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.878471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.878471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.878669 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.878848 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.878849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.878849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.878897 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.879133 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:52.879134 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:52.879169 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:54:52.879222 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:54:52.879294 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:54:52.879475 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.871847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.0.98.61:42341)
0925 14:54:52.872011 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:52.872016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:52.872041 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:54:52.872067 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:52.872069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:52.872084 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:52.872164 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:52.872394 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.873517 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.874172 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.874740 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.874817 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.876697 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:52.876705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:52.876706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:52.876712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:52.878316 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.878738 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.878740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.878741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.878791 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.878949 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.878950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.878950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.879071 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:52.879129 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:52.879242 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 14:54:52.879340 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260925 14:54:52.880132 22747 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:52.880455 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.880356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57446)
0925 14:54:52.880397 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.880438 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:54:52.880615 22697 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
I20260925 14:54:52.880976 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.880918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57454)
0925 14:54:52.880940 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.880962 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 14:54:52.881095 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:54:52.885030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34491
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:52.930374 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } attempt: 1
I20260925 14:54:52.930864 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.930726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57470)
0925 14:54:52.930791 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.930834 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:54:52.931221 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.931123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57482)
0925 14:54:52.931169 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.931194 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:54:52.931317 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
I20260925 14:54:52.931341 22672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } attempt: 3
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:52.938629 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.931649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55554)
0925 14:54:52.931701 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.931712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:52.931754 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:52.931906 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:52.931907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:52.932089 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:52.932161 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.932163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.932826 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.932827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.933639 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:52.933642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.933998 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.933999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:52.934074 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.934551 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:52.934556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:52.935854 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:52.937596 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.937597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.937597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.937754 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.937965 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:52.937965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:52.937970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:52.938026 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:52.938300 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:52.938300 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:52.938379 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 14:54:52.938483 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:54:52.938567 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:54:52.938663 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.931691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.0.98.61:42341)
0925 14:54:52.931728 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:52.931730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:52.931742 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:54:52.931817 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:52.931818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:52.931827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:52.931882 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:52.932232 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.932732 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.933724 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:52.933931 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:52.933979 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:52.935937 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:52.935938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:52.935939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:52.935946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:52.937520 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.937825 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.937827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.937827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.937888 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:52.938089 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:52.938090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:52.938090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:52.938219 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:52.938286 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:52.938494 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 14:54:52.938578 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:52.939131 22672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:52.939460 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.939363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57494)
0925 14:54:52.939409 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.939435 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:54:52.939733 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.939652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57496)
0925 14:54:52.939692 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.939714 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:54:52.939821 22672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
W20260925 14:54:52.969228 22754 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:52.969475 22754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:52.969492 22754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:52.969512 22754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:52.970724 22754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:52.970767 22754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:52.970784 22754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:52.970801 22754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:52.970853 22754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:52.970964 22672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } attempt: 1
I20260925 14:54:52.971424 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.971271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57510)
0925 14:54:52.971343 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.971390 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:54:52.971812 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.971704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57526)
0925 14:54:52.971752 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.971790 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:54:52.971908 22672 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
I20260925 14:54:52.972675 22754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45965,127.0.98.61:42341,127.0.98.60:34491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:52.972926 22754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:52.973151 22754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:52.977988 22760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:52.977981 22759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:52.978067 22762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:52.978219 22754 server_base.cc:1034] running on GCE node
I20260925 14:54:52.978554 22754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:52.978798 22754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:52.980003 22754 hybrid_clock.cc:648] HybridClock initialized: now 1790348092979983 us; error 42 us; skew 500 ppm
Sep 25 14:54:52 dist-test-slave-gpc1 krb5kdc[22591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:52.982476 22754 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:54:52.982872 22754 webserver.cc:460] Webserver started at http://127.0.98.60:45179/ using document root <none> and password file <none>
I20260925 14:54:52.983095 22754 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:52.983165 22754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:52.983314 22754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:52.984364 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data/instance:
uuid: "609bbdfc50e5430894c5c313a99affd7"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "3eab6d98487b79110d2139b82d206577"
server_key_iv: "71bf968eb6b75506d91cc4e477ad6267"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.984737 22754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal/instance:
uuid: "609bbdfc50e5430894c5c313a99affd7"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "3eab6d98487b79110d2139b82d206577"
server_key_iv: "71bf968eb6b75506d91cc4e477ad6267"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.986052 22754 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:54:52.986835 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:52.987004 22754 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:52.987108 22754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "609bbdfc50e5430894c5c313a99affd7"
format_stamp: "Formatted at 2026-09-25 14:54:52 on dist-test-slave-gpc1"
server_key: "3eab6d98487b79110d2139b82d206577"
server_key_iv: "71bf968eb6b75506d91cc4e477ad6267"
server_key_version: "encryptionkey@0"
I20260925 14:54:52.987182 22754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:52.997581 22754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:52.999470 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } attempt: 2
I20260925 14:54:52.999881 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:52.999752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57540)
0925 14:54:52.999807 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:52.999850 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:54:53.000174 22754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:53.000195 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.000100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57546)
0925 14:54:53.000148 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.000171 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:54:53.000294 22747 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34491: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34491: connect: Connection refused (error 111)
I20260925 14:54:53.000391 22754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:53.004320 22754 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34491
I20260925 14:54:53.004393 22821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34491 every 8 connection(s)
I20260925 14:54:53.004714 22754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:54:53.005702 22822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:53.007313 22822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:53.009404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22754
I20260925 14:54:53.009519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347825060285-392-0/minicluster-data/master-2/wal/instance
I20260925 14:54:53.009683 392 external_mini_cluster.cc:1428] Setting key 148147b26251533b270b1392070a4f5d
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:53.016942 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.008065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.98.62:45965)
0925 14:54:53.008130 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.008132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.008309 (+ 177us) server_negotiation.cc:407] Connection header received
0925 14:54:53.008748 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.008749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.008758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.008819 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.009528 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.010201 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.011332 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.011520 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.011604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.013781 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.013782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.013783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.013790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.015211 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.015707 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.015708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.015708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.015758 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.016142 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.016143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.016143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.016327 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.016410 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.016727 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.016804 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:54:53.017067 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.008010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45965 (local address 127.0.0.1:37128)
0925 14:54:53.008183 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.008224 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:53.008671 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:53.008921 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:53.008926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:53.009171 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:53.009399 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.009404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.010297 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.010299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.011221 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.011223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.011632 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.011633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.011748 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.012266 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:53.012275 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:53.013695 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:53.015314 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.015320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.015322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.015627 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.015818 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.015819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.015819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.016004 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.016417 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:53.016424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:53.016588 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 14:54:53.016674 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:54:53.016824 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":25,"thread_start_us":104,"threads_started":1}
I20260925 14:54:53.018103 22822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:53.019279 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.010077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.0.98.60:34491)
0925 14:54:53.010255 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.010260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.010285 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:54:53.010395 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.010398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.010425 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.010539 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.011157 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.011847 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.012594 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.012995 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.013066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.015832 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.015839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.015839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.015846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.018024 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.018475 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.018478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.018479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.018540 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.018753 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.018753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.018754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.018944 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.019060 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.019162 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.019217 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:53.025856 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.018409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42341 (local address 127.0.0.1:55560)
0925 14:54:53.018453 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.018462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:53.018493 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:53.018643 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:53.018644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:53.018795 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:53.018829 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.018831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.019687 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.019688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.020472 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.020474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.021014 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.021015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.021070 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.021423 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:53.021427 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:53.022727 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:53.024790 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.024791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.024791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.024994 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.025175 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.025176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.025177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.025237 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.025568 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:53.025569 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:53.025613 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:54:53.025688 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:54:53.025777 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:54:53.025918 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.018450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.0.98.61:42341)
0925 14:54:53.018491 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.018493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.018501 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:53.018548 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.018549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.018556 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.018630 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.018905 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.019601 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.020550 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.020919 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.020999 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.022813 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.022815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.022815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.022823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.024709 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.025067 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.025068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.025069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.025116 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.025358 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.025359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.025359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.025484 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.025557 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.025728 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.025808 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:53.026254 22822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:53.028041 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.020229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37134 (local address 127.0.98.62:45965)
0925 14:54:53.020312 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.020314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.020355 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:54:53.020413 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.020413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.020422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.020469 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.021028 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.021712 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.022428 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.022666 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.022755 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.025344 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.025345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.025346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.025352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.026913 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.027286 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.027286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.027287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.027333 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.027543 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.027544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.027544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.027667 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.027820 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.027870 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.027941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:53.033495 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.026492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57560)
0925 14:54:53.026531 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.026541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:53.026576 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:53.027101 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:53.027103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:53.027282 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:53.027315 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.027316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.028151 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.028152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.028885 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.028887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.029395 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.029395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.029444 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.029756 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:53.029761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:53.031108 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:53.032620 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.032621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.032622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.032777 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.032972 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.032973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.032973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.033024 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.033283 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:53.033284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:53.033320 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:54:53.033378 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:54:53.033440 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:54:53.033613 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.026547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.98.60:34491)
0925 14:54:53.026991 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.026993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.026998 (+ 5us) server_negotiation.cc:407] Connection header received
0925 14:54:53.027013 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.027014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.027022 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.027058 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.027394 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.028065 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.028956 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.029303 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.029361 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.031204 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.031205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.031206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.031213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.032519 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.032857 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.032858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.032858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.032909 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.033096 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.033097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.033097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.033210 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.033277 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.033425 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.033494 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260925 14:54:53.035415 22822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: Bootstrap starting.
I20260925 14:54:53.036376 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.028650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.0.98.61:42341)
0925 14:54:53.028682 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.028684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.028706 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:54:53.028725 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.028725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.028733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.028785 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.029300 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.029995 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.030913 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.031128 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.031204 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.033583 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.033585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.033585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.033593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.035261 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.035595 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.035596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.035597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.035654 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.035829 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.035830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.035831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.035999 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.036170 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.036206 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.036274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:54:53.036474 22822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:53.036774 22822 log.cc:824] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:53.037670 22822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: No bootstrap required, opened a new log
I20260925 14:54:53.037874 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:53.039441 22822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.038947 22646 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2e63f67b78ce4e37913b42475bca1407"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37134
I20260925 14:54:53.039589 22822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609bbdfc50e5430894c5c313a99affd7, State: Initialized, Role: FOLLOWER
I20260925 14:54:53.039681 22822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.039973 22822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:53.039968 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.040068 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:53.041796 22839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:53.041877 22839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:53.041996 22754 master_runner.cc:186] Error getting master registration for 127.0.98.62:45965: OK, instance_id { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" instance_seqno: 1790348092703925 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:54:53.044744 22754 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:54:53.047048 22672 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } attempt: 2
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:53.054711 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.047305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57564)
0925 14:54:53.047359 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.047369 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:53.047417 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:53.047554 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:53.047555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:53.047737 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:53.047785 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.047787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.048653 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.048654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.049509 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.049511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.049963 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.049964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.050026 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.050548 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:53.050553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:53.052038 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:53.053634 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.053636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.053637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.053812 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.054021 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.054022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.054022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.054087 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.054381 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:53.054382 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:53.054429 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:54:53.054519 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:54:53.054619 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:54:53.054711 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.047340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.60:34491)
0925 14:54:53.047377 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.047379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.047402 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:54:53.047464 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.047465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.047475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.047534 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.047863 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.048568 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.049590 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.049886 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.049948 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.052133 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.052134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.052135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.052141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.053531 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.053894 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.053895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.053895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.053950 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.054158 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.054159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.054159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.054291 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.054386 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.054542 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.054618 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:54:53.056015 22672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: Bootstrap starting.
I20260925 14:54:53.056663 22672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:53.056973 22672 log.cc:824] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:53.057911 22672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: No bootstrap required, opened a new log
I20260925 14:54:53.059021 22672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.059166 22672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e63f67b78ce4e37913b42475bca1407, State: Initialized, Role: FOLLOWER
I20260925 14:54:53.059263 22672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.059623 22672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:53.059589 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.059695 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:53.059746 22646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:54:53.059808 22646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:53.060690 22646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.061709 22796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e63f67b78ce4e37913b42475bca1407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "609bbdfc50e5430894c5c313a99affd7"
I20260925 14:54:53.061841 22796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [term 0 FOLLOWER]: Advancing to term 1
W20260925 14:54:53.061873 22853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:53.061923 22853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:53.062760 22796 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e63f67b78ce4e37913b42475bca1407 in term 1.
I20260925 14:54:53.063313 22622 leader_election.cc:302] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e63f67b78ce4e37913b42475bca1407, 609bbdfc50e5430894c5c313a99affd7; no voters:
I20260925 14:54:53.063429 22842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:53.063596 22842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [term 1 LEADER]: Becoming Leader. State: Replica: 2e63f67b78ce4e37913b42475bca1407, State: Running, Role: LEADER
I20260925 14:54:53.063678 22842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.064101 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e63f67b78ce4e37913b42475bca1407. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.064184 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:53.063818 22604 master_runner.cc:186] Error getting master registration for 127.0.98.61:42341: OK, instance_id { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" instance_seqno: 1790348092857483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:54:53.066056 22604 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:54:53.066334 22646 leader_election.cc:288] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [CANDIDATE]: Term 1 election: Requested vote from peers ed0c2c081c5d439c8c7cc5750e3986fc (127.0.98.61:42341), 609bbdfc50e5430894c5c313a99affd7 (127.0.98.60:34491)
I20260925 14:54:53.070485 22857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:53.070703 22857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:53.075474 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.067467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.0.98.62:45965)
0925 14:54:53.067506 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.067508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.067642 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:54:53.067740 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.067740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.067750 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.067779 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.068589 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.069313 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.070496 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.070774 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.070815 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.072503 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.072505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.072505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.072512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.074181 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.074623 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.074624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.074624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.074684 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.074883 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.074884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.074885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.075002 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.075085 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.075299 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.075385 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:53.079135 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.067767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.0.98.61:42341)
0925 14:54:53.067813 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.067815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.068624 (+ 809us) server_negotiation.cc:407] Connection header received
0925 14:54:53.068779 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.068780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.068792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.068837 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.070157 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.070880 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.072439 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.072739 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.072783 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.075310 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.075311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.075312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.075319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.077200 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.077842 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.077843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.077843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.077906 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.078241 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.078242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.078242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.078406 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.078520 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.078912 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.079010 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:53.079610 22796 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [term 1 FOLLOWER]: Refusing update from remote peer 2e63f67b78ce4e37913b42475bca1407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:53.080014 22842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:53.081844 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e63f67b78ce4e37913b42475bca1407. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.081950 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:53.084095 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.068250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.98.60:34491)
0925 14:54:53.068290 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.068292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.076435 (+ 8143us) server_negotiation.cc:407] Connection header received
0925 14:54:53.076525 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.076526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.076537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.076565 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.077100 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.077850 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.078864 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.079173 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.079219 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.081110 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.081112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.081112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.081119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.082893 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.083271 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.083272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.083272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.083325 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.083534 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.083534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.083535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.083657 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.083722 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.083928 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.084004 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:53.087527 22865 mvcc.cc:205] Tried to move back new op lower bound from 7333265789250387968 to 7333265789189423104. Current Snapshot: MvccSnapshot[applied={T|T < 7333265789250387968}]
I20260925 14:54:53.087618 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.087709 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:53.087656 22828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.087741 22828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:53.088163 22866 mvcc.cc:205] Tried to move back new op lower bound from 7333265789250387968 to 7333265789189423104. Current Snapshot: MvccSnapshot[applied={T|T < 7333265789250387968}]
I20260925 14:54:53.088791 22857 catalog_manager.cc:1347] Generated new cluster ID: 0d2da7ec34fd480a9b6bae6ef0cdc1b6
I20260925 14:54:53.088850 22857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:53.091416 22747 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } attempt: 3
I20260925 14:54:53.096473 22857 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 14:54:53 dist-test-slave-gpc1 krb5kdc[22591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:53.097186 22857 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:53.101964 22750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.091712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34491 (local address 127.0.0.1:57576)
0925 14:54:53.091764 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:53.091775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:53.091819 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:53.092000 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:53.092001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:53.092181 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:53.092229 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.092231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.093150 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.093151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.093973 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:53.093976 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.095120 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.095121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:53.095196 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.095767 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:53.095772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:53.097439 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:53.100508 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.100509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.100510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.100750 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.101208 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:53.101209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:53.101210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:53.101292 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:53.101633 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:53.101634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:53.101696 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 14:54:53.101757 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:54:53.101862 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:54:53.102242 22857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407: Generated new TSK 0
I20260925 14:54:53.102448 22857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:53.102711 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:53.091735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.0.98.60:34491)
0925 14:54:53.091770 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.091772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.091880 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:54:53.091900 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.091900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.091909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.091958 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.092288 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.093033 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.094672 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:53.094984 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:53.095045 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:53.097615 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:53.097616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:53.097616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:53.097637 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:53.100395 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.100848 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.100849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.100849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.100921 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:53.101382 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:53.101383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:53.101383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:53.101529 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:53.101588 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:53.102467 (+ 879us) server_negotiation.cc:299] Negotiation successful
0925 14:54:53.102621 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:53.104223 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: Bootstrap starting.
I20260925 14:54:53.104976 22747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:53.105289 22747 log.cc:824] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:53.107034 22747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: No bootstrap required, opened a new log
I20260925 14:54:53.109030 22747 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.109153 22747 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0c2c081c5d439c8c7cc5750e3986fc, State: Initialized, Role: FOLLOWER
I20260925 14:54:53.109222 22747 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } }
I20260925 14:54:53.109536 22747 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:53.110929 22721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2e63f67b78ce4e37913b42475bca1407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc"
I20260925 14:54:53.111033 22721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:53.111863 22721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e63f67b78ce4e37913b42475bca1407 in term 1.
I20260925 14:54:53.112265 22720 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [term 1 FOLLOWER]: Refusing update from remote peer 2e63f67b78ce4e37913b42475bca1407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:53.112511 22842 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:53.114741 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.114857 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:53.114902 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.114951 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:53.115010 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:53.115067 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:53.115128 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e63f67b78ce4e37913b42475bca1407. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.115320 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:53.127175 22675 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.125696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.98.62:45965)
0925 14:54:53.125734 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.125736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.126541 (+ 805us) server_negotiation.cc:407] Connection header received
0925 14:54:53.126644 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.126646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.126656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.126686 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.127113 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37162: BlockingRecv error: recv got EOF from 127.0.0.1:37162 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:54:53.128028 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.125882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.98.61:42341)
0925 14:54:53.125916 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.125917 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.127530 (+ 1613us) server_negotiation.cc:407] Connection header received
0925 14:54:53.127622 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.127623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.127632 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.127659 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.127958 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55588: BlockingRecv error: recv got EOF from 127.0.0.1:55588 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:54:53.128819 22826 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.126150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.98.60:34491)
0925 14:54:53.126184 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.126186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.128341 (+ 2155us) server_negotiation.cc:407] Connection header received
0925 14:54:53.128437 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.128438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.128449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.128477 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.128750 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57580: BlockingRecv error: recv got EOF from 127.0.0.1:57580 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:54:53.135423 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.134636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.0.98.61:42341)
0925 14:54:53.134677 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.134679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.134929 (+ 250us) server_negotiation.cc:407] Connection header received
0925 14:54:53.135030 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.135031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.135047 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.135075 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.135379 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55592: BlockingRecv error: recv got EOF from 127.0.0.1:55592 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 14:54:53.138454 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.137689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.0.98.61:42341)
0925 14:54:53.137721 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.137722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.137968 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:54:53.138052 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.138053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.138062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.138089 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.138414 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55602: BlockingRecv error: recv got EOF from 127.0.0.1:55602 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:54:53.142437 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.141618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.98.61:42341)
0925 14:54:53.141648 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.141650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.141906 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:54:53.142004 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.142005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.142014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.142042 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.142326 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55616: BlockingRecv error: recv got EOF from 127.0.0.1:55616 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 14:54:53.148407 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.147629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.98.61:42341)
0925 14:54:53.147660 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.147662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.147910 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:54:53.148026 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.148027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.148037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.148067 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.148368 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55622: BlockingRecv error: recv got EOF from 127.0.0.1:55622 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:54:53.158553 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.157645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.0.98.61:42341)
0925 14:54:53.157685 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.157687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.157970 (+ 283us) server_negotiation.cc:407] Connection header received
0925 14:54:53.158074 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.158075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.158087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.158117 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.158502 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55624: BlockingRecv error: recv got EOF from 127.0.0.1:55624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:54:53.176913 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.175971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.98.61:42341)
0925 14:54:53.176017 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.176020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.176313 (+ 293us) server_negotiation.cc:407] Connection header received
0925 14:54:53.176419 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.176421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.176432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.176463 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.176851 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55632: BlockingRecv error: recv got EOF from 127.0.0.1:55632 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:54:53.211225 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.210311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.0.98.61:42341)
0925 14:54:53.210375 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.210378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.210654 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:54:53.210755 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.210756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.210768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.210808 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.211165 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55642: BlockingRecv error: recv got EOF from 127.0.0.1:55642 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:54:53.277381 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.276691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.0.98.61:42341)
0925 14:54:53.276751 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.276754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.276904 (+ 150us) server_negotiation.cc:407] Connection header received
0925 14:54:53.276936 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.276936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.276949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.277006 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.277320 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55644: BlockingRecv error: recv got EOF from 127.0.0.1:55644 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 14:54:53.407215 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.406434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.0.98.61:42341)
0925 14:54:53.406528 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.406530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.406616 (+ 86us) server_negotiation.cc:407] Connection header received
0925 14:54:53.406716 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.406717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.406729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.406799 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.407150 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55654: BlockingRecv error: recv got EOF from 127.0.0.1:55654 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 14:54:53.664991 22752 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:53.664319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.0.98.61:42341)
0925 14:54:53.664391 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:53.664393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:53.664416 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:54:53.664487 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:53.664488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:53.664499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:53.664540 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:53.664907 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55664: BlockingRecv error: recv got EOF from 127.0.0.1:55664 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:53.687983 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e63f67b78ce4e37913b42475bca1407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e63f67b78ce4e37913b42475bca1407" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45965 } } peers { permanent_uuid: "ed0c2c081c5d439c8c7cc5750e3986fc" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42341 } } peers { permanent_uuid: "609bbdfc50e5430894c5c313a99affd7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34491 } } }
I20260925 14:54:53.688161 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e63f67b78ce4e37913b42475bca1407 [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:54.042320 22839 catalog_manager.cc:1259] Loaded cluster ID: 0d2da7ec34fd480a9b6bae6ef0cdc1b6
I20260925 14:54:54.042454 22839 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: loading cluster ID for follower catalog manager: success
I20260925 14:54:54.043221 22839 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: acquiring CA information for follower catalog manager: success
I20260925 14:54:54.043458 22839 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 609bbdfc50e5430894c5c313a99affd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:54:54.115481 22882 catalog_manager.cc:1259] Loaded cluster ID: 0d2da7ec34fd480a9b6bae6ef0cdc1b6
I20260925 14:54:54.115573 22882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: loading cluster ID for follower catalog manager: success
I20260925 14:54:54.116478 22882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: acquiring CA information for follower catalog manager: success
I20260925 14:54:54.116798 22882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed0c2c081c5d439c8c7cc5750e3986fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:54:54.180694 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:54.178112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.98.61:42341)
0925 14:54:54.178257 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:54.178259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:54.178273 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:54:54.178394 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:54.178394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:54.178454 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:54.178563 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:54.178745 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.179262 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.179989 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.180176 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.180225 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:54.180360 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:54.180467 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:54.180524 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:54.180561 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:54:54.180578 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260925 14:54:54.184998 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:54.182522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.0.98.60:34491)
0925 14:54:54.182619 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:54.182621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:54.182630 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:54:54.182717 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:54.182718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:54.182770 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:54.182856 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:54.182968 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.183450 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.184166 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.184420 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.184476 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:54.184653 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:54.184765 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:54.184853 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:54.184890 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:54:54.184911 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
I20260925 14:54:54.189481 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:54.186601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.98.62:45965)
0925 14:54:54.186695 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:54.186698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:54.186709 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:54:54.186801 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:54.186802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:54.186852 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:54.186929 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:54.187109 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.187589 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.188529 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.188826 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.188899 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:54.189142 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:54.189233 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:54.189302 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:54.189360 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:54:54.189382 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":51,"threads_started":1}
I20260925 14:54:54.189831 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:54.186604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.0.98.61:42341)
0925 14:54:54.186696 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:54.186698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:54.186806 (+ 108us) server_negotiation.cc:407] Connection header received
0925 14:54:54.186869 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:54.186870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:54.186879 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:54.186929 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:54.187100 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.187849 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.188965 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.189241 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.189293 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:54.189520 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:54.189550 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:54.189577 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:54.189739 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:54:54.189767 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:54:54.189968 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:54.186873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57610 (local address 127.0.98.60:34491)
0925 14:54:54.186916 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:54.186918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:54.187091 (+ 173us) server_negotiation.cc:407] Connection header received
0925 14:54:54.187183 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:54.187184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:54.187193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:54.187219 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:54.187383 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.188042 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.189123 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:54.189445 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:54.189500 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:54.189708 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:54.189738 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:54.189766 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:54.189852 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:54:54.189873 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:54.191218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22604
I20260925 14:54:54.196732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22679
I20260925 14:54:54.201613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22754
2026-09-25T14:54:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2772 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:54:54 dist-test-slave-gpc1 krb5kdc[22975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:54:54 dist-test-slave-gpc1 krb5kdc[22975](info): set up 2 sockets
Sep 25 14:54:54 dist-test-slave-gpc1 krb5kdc[22975](info): commencing operation
krb5kdc: starting...
W20260925 14:54:55.250661 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/test-user.keytab.
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:54:55.360412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45869
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:54:55.440043 22988 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:55.440279 22988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:55.440317 22988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:55.440358 22988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:55.441495 22988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:55.441568 22988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:55.441591 22988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:55.441622 22988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:55.441648 22988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:55.443229 22988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:55.443557 22988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:55.443794 22988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:55.448709 22996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:55.448720 22993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:55.448997 22988 server_base.cc:1034] running on GCE node
W20260925 14:54:55.448724 22994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:55.449265 22988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:55.449537 22988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:55.450701 22988 hybrid_clock.cc:648] HybridClock initialized: now 1790348095450696 us; error 36 us; skew 500 ppm
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:55.453109 22988 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:54:55.453497 22988 webserver.cc:460] Webserver started at http://127.0.98.62:37915/ using document root <none> and password file <none>
I20260925 14:54:55.453688 22988 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:55.453728 22988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:55.453811 22988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:55.454896 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data/instance:
uuid: "b7eaba947f814e5cba562c2838fcabdc"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d357662200c2adebb2714f948c885864"
server_key_iv: "fe6e0a1255feff65e72af11a8b2cb4ce"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.455193 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7eaba947f814e5cba562c2838fcabdc"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d357662200c2adebb2714f948c885864"
server_key_iv: "fe6e0a1255feff65e72af11a8b2cb4ce"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.456384 22988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:54:55.457048 23003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:55.457209 22988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:55.457289 22988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal
uuid: "b7eaba947f814e5cba562c2838fcabdc"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d357662200c2adebb2714f948c885864"
server_key_iv: "fe6e0a1255feff65e72af11a8b2cb4ce"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.457343 22988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:55.469677 22988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:55.471621 22988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:55.471732 22988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:55.475605 22988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45869
I20260925 14:54:55.475672 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45869 every 8 connection(s)
I20260925 14:54:55.475999 22988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:54:55.476989 23056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:55.478564 23056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:55.484956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 22988
I20260925 14:54:55.485065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-0/wal/instance
I20260925 14:54:55.485282 392 external_mini_cluster.cc:1428] Setting key f97d4c082ae887c1985b65bea6a2724e
I20260925 14:54:55.488677 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.479265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45869 (local address 127.0.0.1:38960)
0925 14:54:55.479423 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.479469 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.479968 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.480252 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.480254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.480560 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.480708 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.480712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.481727 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.481728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.482420 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.482422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.482920 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.482921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.482982 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.483481 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.483489 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.485206 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.487228 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.487234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.487235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.487420 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.487680 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.487681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.487681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.487761 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.488106 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.488111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.488187 (+ 76us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.488283 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.488506 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":91,"thread_start_us":74,"threads_started":1}
I20260925 14:54:55.488682 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.479312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.98.62:45869)
0925 14:54:55.479473 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.479480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.479595 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:54:55.480128 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.480130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.480154 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.480247 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.480821 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.481620 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.482508 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.482809 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.482933 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.485314 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.485350 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.485351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.485360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.487149 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.487528 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.487530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.487531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.487613 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.487827 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.487828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.487828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.488029 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.488114 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.488306 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.488446 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260925 14:54:55.489336 23056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:55.489673 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.489558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48364)
0925 14:54:55.489604 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.489644 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:54:55.489789 23006 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
I20260925 14:54:55.490203 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.490114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48380)
0925 14:54:55.490161 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.490184 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:54:55.490305 23056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:54:55.509491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40853
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:55.539523 23056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } attempt: 1
I20260925 14:54:55.540012 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.539878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48382)
0925 14:54:55.539941 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.539984 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:54:55.540400 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.540304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48392)
0925 14:54:55.540351 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.540375 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:54:55.540488 23056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
W20260925 14:54:55.590555 23063 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:55.590811 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:55.590844 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:55.590891 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:55.592011 23063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:55.592089 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:55.592116 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:55.592149 23063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:55.592175 23063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:55.593621 23063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:55.593883 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:55.594102 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:55.598837 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:55.598809 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:55.598816 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:55.599287 23063 server_base.cc:1034] running on GCE node
I20260925 14:54:55.599574 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:55.599893 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:55.601061 23063 hybrid_clock.cc:648] HybridClock initialized: now 1790348095601046 us; error 28 us; skew 500 ppm
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:55.603754 23063 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:54:55.604281 23063 webserver.cc:460] Webserver started at http://127.0.98.61:39131/ using document root <none> and password file <none>
I20260925 14:54:55.604564 23063 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:55.604640 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:55.604799 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:55.605894 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data/instance:
uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d078c7d8265bc388d24524663f5e12df"
server_key_iv: "dc6602c42b3bc7ab2cf592b702d71309"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.606289 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal/instance:
uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d078c7d8265bc388d24524663f5e12df"
server_key_iv: "dc6602c42b3bc7ab2cf592b702d71309"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.607761 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:54:55.608677 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:55.608644 23056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } attempt: 2
I20260925 14:54:55.608889 23063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:54:55.608989 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal
uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "d078c7d8265bc388d24524663f5e12df"
server_key_iv: "dc6602c42b3bc7ab2cf592b702d71309"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.609088 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.608926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48404)
0925 14:54:55.608989 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.609024 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 14:54:55.609072 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:55.609570 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.609461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48418)
0925 14:54:55.609511 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.609541 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:54:55.609716 23056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40853: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40853: connect: Connection refused (error 111)
I20260925 14:54:55.621731 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:55.623718 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:55.623832 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:55.627815 23063 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40853
I20260925 14:54:55.627876 23130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40853 every 8 connection(s)
I20260925 14:54:55.628185 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:54:55.629134 23131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:55.630803 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:55.634231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 23063
I20260925 14:54:55.634356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-1/wal/instance
I20260925 14:54:55.634600 392 external_mini_cluster.cc:1428] Setting key fa52edf20c71e9a2f86f0e4c157438f5
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:55.639588 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.631640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.0.98.62:45869)
0925 14:54:55.631690 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.631693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.631728 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:54:55.632217 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.632218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.632233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.632284 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.632851 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.633325 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.634286 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.634600 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.634681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.636759 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.636761 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.636761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.636769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.638201 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.638657 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.638658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.638658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.638717 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.638972 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.638973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.638974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.639088 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.639163 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.639418 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.639500 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:54:55.639780 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.631433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45869 (local address 127.0.0.1:38976)
0925 14:54:55.631619 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.631658 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.632112 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.632352 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.632355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.632580 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.632758 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.632762 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.633429 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.633430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.634188 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.634189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.634710 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.634711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.634828 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.635330 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.635338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.636665 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.638302 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.638309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.638311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.638567 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.638784 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.638785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.638785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.638904 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.639155 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.639161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.639295 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.639397 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.639547 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":111,"threads_started":1}
I20260925 14:54:55.640431 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:55.648061 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.640695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48426)
0925 14:54:55.640735 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.640761 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.640794 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.641057 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.641057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.641194 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.641233 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.641235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.642124 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.642124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.642703 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.642705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.643189 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.643190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.643250 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.643661 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.643665 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.644899 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.646908 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.646909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.646910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.647101 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.647329 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.647330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.647331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.647389 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.647700 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.647701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.647743 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.647813 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.647923 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:54:55.648145 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.640786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.0.98.61:40853)
0925 14:54:55.640933 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.640937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.640952 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:54:55.640968 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.640970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.640980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.641058 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.641314 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.642043 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.642792 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.643104 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.643190 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.645000 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.645007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.645008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.645014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.646800 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.647193 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.647197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.647197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.647259 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.647469 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.647470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.647470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.647625 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.647723 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.647843 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.647996 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260925 14:54:55.648726 23131 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:55.649091 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.648980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35082)
0925 14:54:55.649029 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.649072 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 14:54:55.649272 23081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
I20260925 14:54:55.649729 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.649660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35096)
0925 14:54:55.649689 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.649713 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 14:54:55.649859 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:54:55.659713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbEnjDj/build/release/bin/kudu
/tmp/dist-test-taskbEnjDj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46383
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:54:55.699136 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } attempt: 1
I20260925 14:54:55.699672 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.699526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35110)
0925 14:54:55.699600 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.699642 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:54:55.700114 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.700024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35126)
0925 14:54:55.700073 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.700098 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:54:55.700238 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
I20260925 14:54:55.700873 23056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } attempt: 3
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:55.708312 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.701211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48442)
0925 14:54:55.701298 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.701309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.701368 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.701524 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.701525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.701709 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.701785 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.701787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.702457 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.702458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.703279 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.703282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.703630 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.703632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.703710 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.704120 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.704125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.705430 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.707331 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.707332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.707332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.707493 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.707722 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.707723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.707723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.707775 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.708073 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.708074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.708111 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.708161 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.708246 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 14:54:55.708374 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.701256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.0.98.61:40853)
0925 14:54:55.701306 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.701308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.701349 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:54:55.701429 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.701430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.701440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.701493 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.701858 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.702336 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.703358 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.703566 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.703617 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.705509 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.705510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.705511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.705518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.707243 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.707576 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.707578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.707578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.707650 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.707840 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.707841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.707841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.708004 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.708081 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.708199 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.708281 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:54:55.709044 23056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:55.709411 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.709312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35142)
0925 14:54:55.709358 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.709388 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:54:55.709769 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.709685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35146)
0925 14:54:55.709730 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.709753 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:54:55.709887 23056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
I20260925 14:54:55.741091 23056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } attempt: 1
I20260925 14:54:55.741636 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.741480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35154)
0925 14:54:55.741548 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.741597 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:54:55.741987 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.741891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35160)
0925 14:54:55.741919 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.741946 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 14:54:55.742081 23056 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
W20260925 14:54:55.749219 23138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:54:55.749410 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:54:55.749428 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:54:55.749447 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:54:55.750566 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:54:55.750604 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:54:55.750618 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:54:55.750630 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:54:55.750643 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:54:55.752058 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45869,127.0.98.61:40853,127.0.98.60:46383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:55:42 UTC on ccycloud.quasar-zoybvp.root.comops.site
build id 4672723
I20260925 14:54:55.752235 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:54:55.752398 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:54:55.757189 23146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:54:55.757155 23144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:55.757339 23138 server_base.cc:1034] running on GCE node
W20260925 14:54:55.757349 23143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:54:55.757660 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:54:55.757932 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:54:55.759101 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790348095759096 us; error 36 us; skew 500 ppm
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:54:55.761534 23138 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:54:55.761991 23138 webserver.cc:460] Webserver started at http://127.0.98.60:46753/ using document root <none> and password file <none>
I20260925 14:54:55.762216 23138 fs_manager.cc:359] Metadata directory not provided
I20260925 14:54:55.762282 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:54:55.762465 23138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:54:55.763489 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data/instance:
uuid: "ef594d7772c64d8c8e188e24bd91ee6b"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "36f3e383f7d4e988edb8b147f68da954"
server_key_iv: "99c422fcc7ef1aa1d7058ed00ad54ad3"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.763836 23138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal/instance:
uuid: "ef594d7772c64d8c8e188e24bd91ee6b"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "36f3e383f7d4e988edb8b147f68da954"
server_key_iv: "99c422fcc7ef1aa1d7058ed00ad54ad3"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.765153 23138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:54:55.765848 23153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:54:55.766016 23138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:54:55.766117 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal
uuid: "ef594d7772c64d8c8e188e24bd91ee6b"
format_stamp: "Formatted at 2026-09-25 14:54:55 on dist-test-slave-gpc1"
server_key: "36f3e383f7d4e988edb8b147f68da954"
server_key_iv: "99c422fcc7ef1aa1d7058ed00ad54ad3"
server_key_version: "encryptionkey@0"
I20260925 14:54:55.766188 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:54:55.768385 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } attempt: 2
I20260925 14:54:55.768929 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.768764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35174)
0925 14:54:55.768833 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.768891 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:54:55.769310 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.769163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35180)
0925 14:54:55.769225 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.769257 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 14:54:55.769793 23131 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46383: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46383: connect: Connection refused (error 111)
I20260925 14:54:55.784776 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:54:55.786777 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:54:55.786932 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:54:55.791069 23138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46383
I20260925 14:54:55.791116 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46383 every 8 connection(s)
I20260925 14:54:55.791518 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:54:55.792478 23206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:54:55.793968 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:54:55.794819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbEnjDj/build/release/bin/kudu as pid 23138
I20260925 14:54:55.794940 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbEnjDj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347825060285-392-0/minicluster-data/master-2/wal/instance
I20260925 14:54:55.795176 392 external_mini_cluster.cc:1428] Setting key 1cd9c9a9ddfec3a2c7929b6ddca7837e
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:55.804338 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.794740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.0.98.62:45869)
0925 14:54:55.794813 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.794815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.795044 (+ 229us) server_negotiation.cc:407] Connection header received
0925 14:54:55.795811 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.795812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.795827 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.795877 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.796539 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.797065 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.798035 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.798530 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.798631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.800940 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.800942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.800942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.800949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.802881 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.803334 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.803336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.803336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.803398 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.803672 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.803673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.803673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.803796 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.803876 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.804164 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.804247 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:54:55.804737 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.795520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35196 (local address 127.0.98.60:46383)
0925 14:54:55.795807 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.795812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.795839 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:55.795900 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.795901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.795916 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.796009 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.796664 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.797459 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.798204 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.798759 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.798853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.801530 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.801538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.801538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.801545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.803498 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.803827 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.803829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.803830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.803886 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.804087 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.804088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.804089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.804231 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.804495 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.804543 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.804615 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":70,"threads_started":1}
I20260925 14:54:55.804747 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.794682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45869 (local address 127.0.0.1:38980)
0925 14:54:55.794861 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.794913 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.795673 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.795915 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.795920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.796227 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.796399 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.796405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.797168 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.797169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.797928 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.797930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.798662 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.798664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.798793 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.799331 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.799341 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.800860 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.802969 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.802975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.802977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.803243 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.803475 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.803476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.803476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.803584 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.803865 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.803872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.804007 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.804112 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.804274 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260925 14:54:55.805287 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:54:55.812671 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.805535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40853 (local address 127.0.0.1:48456)
0925 14:54:55.805739 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.805750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.805802 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.805942 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.805943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.806099 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.806136 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.806138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.806978 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.806979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.807673 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.807675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.808056 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.808057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.808098 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.808476 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.808481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.809824 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.811603 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.811604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.811605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.811827 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.812041 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.812042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.812042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.812099 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.812352 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.812353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.812395 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.812482 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.812579 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260925 14:54:55.812671 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.805592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.0.98.61:40853)
0925 14:54:55.805634 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.805636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.805804 (+ 168us) server_negotiation.cc:407] Connection header received
0925 14:54:55.805868 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.805869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.805880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.805941 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.806208 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.806899 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.807753 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.807970 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.808044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.809909 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.809910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.809911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.809917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.811503 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.811910 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.811912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.811912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.811968 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.812172 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.812173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.812173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.812286 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.812359 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.812507 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.812579 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:55.813258 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.805737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.0.98.62:45869)
0925 14:54:55.805767 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.805769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.805820 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:54:55.805881 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.805882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.805891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.805941 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.806545 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.807148 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.807984 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.808257 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.808319 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.810582 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.810583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.810583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.810590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.812024 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.812414 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.812415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.812415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.812474 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.812673 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.812674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.812674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.812813 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.812981 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.813016 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.813086 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:55.813506 23206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:55.817214 23056 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } attempt: 2
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:54:55.820634 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.813775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35212)
0925 14:54:55.813814 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.813822 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.813856 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.813969 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.813971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.814094 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.814115 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.814116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.814817 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.814818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.815278 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.815280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.815665 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.815666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.815713 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.816117 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.816122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.817377 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.819528 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.819529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.819530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.819729 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.819957 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.819957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.819958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.820024 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.820314 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.820315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.820358 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.820425 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.820512 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 14:54:55.820887 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.813824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.0.98.60:46383)
0925 14:54:55.813856 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.813858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.813865 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:55.813905 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.813906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.813916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.813959 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.814193 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.814727 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.815370 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.815578 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.815665 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.817442 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.817443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.817444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.817450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.819445 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.819800 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.819801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.819801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.819876 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.820106 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.820107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.820108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.820248 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.820310 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.820752 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.820819 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:55.821700 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.813905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.0.98.61:40853)
0925 14:54:55.813940 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.813942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.813950 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:54:55.814011 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.814012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.814021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.814067 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.814575 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.815003 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.815660 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.815929 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.815997 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.818260 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.818261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.818261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.818269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.820183 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.820549 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.820550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.820551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.820602 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.820912 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.820913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.820914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.821307 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.821495 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.821530 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.821597 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:55.822650 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: Bootstrap starting.
I20260925 14:54:55.823175 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:54:55.823451 23206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:55.823802 23206 log.cc:824] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:55.824738 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: No bootstrap required, opened a new log
I20260925 14:54:55.825198 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.817455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35220)
0925 14:54:55.817495 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.817505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.817540 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.817903 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.817904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.818067 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.818109 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.818111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.818853 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.818854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.819508 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.819510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.820384 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.820385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.820443 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.820944 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.820950 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.822260 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.824116 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.824117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.824118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.824349 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.824558 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.824559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.824559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.824615 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.824891 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.824892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.824935 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.825002 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.825097 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:54:55.824267 23030 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b7eaba947f814e5cba562c2838fcabdc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38992
I20260925 14:54:55.825529 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.817549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.60:46383)
0925 14:54:55.817744 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.817747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.817754 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:54:55.817774 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.817775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.817785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.817891 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.818189 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.818772 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.819613 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.820328 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.820383 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.822399 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.822401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.822401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.822407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.824049 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.824422 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.824424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.824424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.824477 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.824693 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.824694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.824695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.824820 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.824898 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.825330 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.825405 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":122,"threads_started":1}
I20260925 14:54:55.826861 23206 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.826990 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: Bootstrap starting.
I20260925 14:54:55.827014 23206 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef594d7772c64d8c8e188e24bd91ee6b, State: Initialized, Role: FOLLOWER
I20260925 14:54:55.827107 23206 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.827369 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.827430 23206 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:55.827486 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.827672 23056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:55.827988 23056 log.cc:824] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:55.828847 23056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: No bootstrap required, opened a new log
W20260925 14:54:55.829430 23225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:55.829509 23225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:55.830286 23056 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.830439 23056 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7eaba947f814e5cba562c2838fcabdc, State: Initialized, Role: FOLLOWER
I20260925 14:54:55.830514 23056 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.830853 23030 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:54:55.830927 23030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:55.829643 23138 master_runner.cc:186] Error getting master registration for 127.0.98.62:45869: OK, instance_id { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" instance_seqno: 1790348095474479 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:54:55.831642 23030 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.831715 23138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:54:55.831849 23030 leader_election.cc:288] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [CANDIDATE]: Term 1 election: Requested vote from peers 94ac2c3de8ba496696c85dc0fc6ffcdf (127.0.98.61:40853), ef594d7772c64d8c8e188e24bd91ee6b (127.0.98.60:46383)
I20260925 14:54:55.832199 23180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7eaba947f814e5cba562c2838fcabdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef594d7772c64d8c8e188e24bd91ee6b"
I20260925 14:54:55.832301 23180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:55.830806 23056 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:55.833154 23180 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7eaba947f814e5cba562c2838fcabdc in term 1.
I20260925 14:54:55.833365 23006 leader_election.cc:302] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7eaba947f814e5cba562c2838fcabdc, ef594d7772c64d8c8e188e24bd91ee6b; no voters:
I20260925 14:54:55.833585 23237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:54:55.833745 23237 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [term 1 LEADER]: Becoming Leader. State: Replica: b7eaba947f814e5cba562c2838fcabdc, State: Running, Role: LEADER
I20260925 14:54:55.833844 23237 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.830801 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.834064 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.834295 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7eaba947f814e5cba562c2838fcabdc. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.834749 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: This master's current role is: LEADER
W20260925 14:54:55.835330 23245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:54:55.835384 23245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:55.835887 23247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:54:55.836094 23247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:54:55.835511 22988 master_runner.cc:186] Error getting master registration for 127.0.98.61:40853: OK, instance_id { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" instance_seqno: 1790348095626678 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:54:55.837671 22988 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:54:55.839327 23180 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [term 1 FOLLOWER]: Refusing update from remote peer b7eaba947f814e5cba562c2838fcabdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:55.839679 23239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:55.840955 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7eaba947f814e5cba562c2838fcabdc. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.841044 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.843251 23239 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.843341 23239 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:55.843823 23213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.843906 23213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.844203 23250 mvcc.cc:205] Tried to move back new op lower bound from 7333265800554848256 to 7333265800536129536. Current Snapshot: MvccSnapshot[applied={T|T < 7333265800554848256}]
I20260925 14:54:55.845181 23251 mvcc.cc:205] Tried to move back new op lower bound from 7333265800554848256 to 7333265800536129536. Current Snapshot: MvccSnapshot[applied={T|T < 7333265800554848256}]
I20260925 14:54:55.845965 23247 catalog_manager.cc:1347] Generated new cluster ID: 3b13775f74b84e0bbcfa8d567f054c0d
I20260925 14:54:55.846015 23247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:54:55.852299 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.837931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.0.98.60:46383)
0925 14:54:55.837977 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.837980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.839479 (+ 1499us) server_negotiation.cc:407] Connection header received
0925 14:54:55.839568 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.839569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.839578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.839607 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.842130 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.842869 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.844616 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.844995 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.845037 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.847636 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.847637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.847637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.847644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.849510 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.850867 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.850868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.850869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.850919 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.851605 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.851606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.851606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.851732 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.851823 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.852123 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.852209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":24}
I20260925 14:54:55.852643 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.834405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.61:40853)
0925 14:54:55.834444 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.834445 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.838417 (+ 3972us) server_negotiation.cc:407] Connection header received
0925 14:54:55.838562 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.838563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.838575 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.838606 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.840344 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.841080 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.843895 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.844219 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.844261 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.847925 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.847926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.847926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.847933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.850225 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.851129 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.851130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.851130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.851178 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.851354 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.851355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.851355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.851474 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.851551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.852476 (+ 925us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.852552 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:55.860894 23131 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } attempt: 3
I20260925 14:54:55.861063 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.845605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.98.62:45869)
0925 14:54:55.845646 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.845648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.853796 (+ 8148us) server_negotiation.cc:407] Connection header received
0925 14:54:55.853903 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.853904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.853916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.853945 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.854626 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.855362 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.856324 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.856624 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.856663 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.858113 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.858114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.858115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.858122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.859813 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.860223 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.860224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.860224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.860276 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.860482 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.860483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.860483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.860598 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.860679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.860874 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.860965 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:54:55.863538 23247 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 14:54:55 dist-test-slave-gpc1 krb5kdc[22975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:54:55.865839 23247 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:54:55.868796 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.861166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46383 (local address 127.0.0.1:35228)
0925 14:54:55.861243 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:54:55.861254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:54:55.861306 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:54:55.861476 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:54:55.861477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:54:55.861628 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:54:55.861668 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.861669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.862590 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.862591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.863393 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:54:55.863395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.863887 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.863888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:54:55.863944 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.864521 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:54:55.864526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:54:55.865921 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:54:55.867762 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.867763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.867764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.867965 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.868166 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:54:55.868167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:54:55.868167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:54:55.868240 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:54:55.868506 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:54:55.868507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:54:55.868548 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:54:55.868609 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:54:55.868699 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 14:54:55.869132 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:55.861210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.98.60:46383)
0925 14:54:55.861243 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.861245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.861292 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:54:55.861365 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.861365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.861376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.861432 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.861743 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.862500 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.863482 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:55.863785 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:55.863841 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:55.866011 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:54:55.866012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:54:55.866013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:54:55.866019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:54:55.867670 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.868037 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.868038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.868038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.868100 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:54:55.868307 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:54:55.868308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:54:55.868308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:54:55.868427 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:54:55.868482 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:55.868961 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 14:54:55.869033 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:54:55.870798 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: Bootstrap starting.
I20260925 14:54:55.871527 23131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: Neither blocks nor log segments found. Creating new log.
I20260925 14:54:55.871825 23131 log.cc:824] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:54:55.878247 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: No bootstrap required, opened a new log
I20260925 14:54:55.880393 23131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.880556 23131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ac2c3de8ba496696c85dc0fc6ffcdf, State: Initialized, Role: FOLLOWER
I20260925 14:54:55.880645 23131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } }
I20260925 14:54:55.880995 23131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:54:55.881067 23105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7eaba947f814e5cba562c2838fcabdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf"
I20260925 14:54:55.881207 23105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:54:55.882099 23105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7eaba947f814e5cba562c2838fcabdc in term 1.
I20260925 14:54:55.882329 23104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [term 1 FOLLOWER]: Refusing update from remote peer b7eaba947f814e5cba562c2838fcabdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:54:55.883901 23237 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:54:55.886452 23247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc: Generated new TSK 0
I20260925 14:54:55.886650 23247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:54:55.886847 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.886934 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:54:55.886958 23269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 14:54:55.886987 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7eaba947f814e5cba562c2838fcabdc. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
W20260925 14:54:55.887006 23269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:54:55.887041 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.887080 23273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.887130 23273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:54:55.887823 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7eaba947f814e5cba562c2838fcabdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7eaba947f814e5cba562c2838fcabdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45869 } } peers { permanent_uuid: "94ac2c3de8ba496696c85dc0fc6ffcdf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40853 } } peers { permanent_uuid: "ef594d7772c64d8c8e188e24bd91ee6b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46383 } } }
I20260925 14:54:55.887920 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7eaba947f814e5cba562c2838fcabdc [sys.catalog]: This master's current role is: LEADER
I20260925 14:54:55.888077 23271 mvcc.cc:205] Tried to move back new op lower bound from 7333265800647766016 to 7333265800536129536. Current Snapshot: MvccSnapshot[applied={T|T < 7333265800554848256}]
W20260925 14:54:55.917070 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.915774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.0.98.61:40853)
0925 14:54:55.915817 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.915819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.916408 (+ 589us) server_negotiation.cc:407] Connection header received
0925 14:54:55.916514 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.916515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.916526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.916556 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.916991 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48484: BlockingRecv error: recv got EOF from 127.0.0.1:48484 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:54:55.918023 23211 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.916123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.0.98.60:46383)
0925 14:54:55.916162 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.916164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.917517 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 14:54:55.917608 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.917609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.917619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.917647 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.917963 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35234: BlockingRecv error: recv got EOF from 127.0.0.1:35234 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:54:55.918784 23059 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.917405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.98.62:45869)
0925 14:54:55.917443 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.917445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.918211 (+ 766us) server_negotiation.cc:407] Connection header received
0925 14:54:55.918384 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.918385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.918396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.918425 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.918729 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39008: BlockingRecv error: recv got EOF from 127.0.0.1:39008 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:54:55.921229 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.920397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.61:40853)
0925 14:54:55.920444 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.920446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.920662 (+ 216us) server_negotiation.cc:407] Connection header received
0925 14:54:55.920769 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.920770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.920788 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.920818 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.921182 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48498: BlockingRecv error: recv got EOF from 127.0.0.1:48498 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:54:55.924230 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.923418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.0.98.61:40853)
0925 14:54:55.923451 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.923453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.923704 (+ 251us) server_negotiation.cc:407] Connection header received
0925 14:54:55.923815 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.923816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.923827 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.923858 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.924183 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48504: BlockingRecv error: recv got EOF from 127.0.0.1:48504 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:54:55.928128 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.927252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.0.98.61:40853)
0925 14:54:55.927294 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.927296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.927595 (+ 299us) server_negotiation.cc:407] Connection header received
0925 14:54:55.927701 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.927702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.927712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.927744 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.928088 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48508: BlockingRecv error: recv got EOF from 127.0.0.1:48508 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:54:55.934156 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.933237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.0.98.61:40853)
0925 14:54:55.933271 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.933273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.933635 (+ 362us) server_negotiation.cc:407] Connection header received
0925 14:54:55.933735 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.933736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.933748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.933779 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.934111 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48510: BlockingRecv error: recv got EOF from 127.0.0.1:48510 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:54:55.944183 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.943312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48512 (local address 127.0.98.61:40853)
0925 14:54:55.943348 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.943350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.943616 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:54:55.943721 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.943722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.943733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.943764 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.944133 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48512: BlockingRecv error: recv got EOF from 127.0.0.1:48512 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:54:55.962476 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.961492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48522 (local address 127.0.98.61:40853)
0925 14:54:55.961531 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.961533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.961824 (+ 291us) server_negotiation.cc:407] Connection header received
0925 14:54:55.961935 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.961936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.961949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.961990 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.962422 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48522: BlockingRecv error: recv got EOF from 127.0.0.1:48522 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:54:55.996886 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:55.995852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48532 (local address 127.0.98.61:40853)
0925 14:54:55.995907 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:55.995910 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:55.996284 (+ 374us) server_negotiation.cc:407] Connection header received
0925 14:54:55.996390 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:55.996391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:55.996403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:55.996434 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:55.996824 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48532: BlockingRecv error: recv got EOF from 127.0.0.1:48532 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:54:56.062973 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:56.062130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.61:40853)
0925 14:54:56.062195 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.062197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.062321 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:54:56.062453 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.062455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.062474 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.062542 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:56.062905 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48540: BlockingRecv error: recv got EOF from 127.0.0.1:48540 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 14:54:56.192848 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:56.192223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.0.98.61:40853)
0925 14:54:56.192299 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.192301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.192321 (+ 20us) server_negotiation.cc:407] Connection header received
0925 14:54:56.192386 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.192387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.192398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.192459 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:56.192772 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48546: BlockingRecv error: recv got EOF from 127.0.0.1:48546 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 14:54:56.450735 23135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:54:56.450062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.0.98.61:40853)
0925 14:54:56.450139 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.450142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.450169 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:54:56.450198 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.450199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.450209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.450300 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:54:56.450671 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48550: BlockingRecv error: recv got EOF from 127.0.0.1:48550 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:54:56.829952 23225 catalog_manager.cc:1259] Loaded cluster ID: 3b13775f74b84e0bbcfa8d567f054c0d
I20260925 14:54:56.830029 23225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: loading cluster ID for follower catalog manager: success
I20260925 14:54:56.830880 23225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: acquiring CA information for follower catalog manager: success
I20260925 14:54:56.831204 23225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef594d7772c64d8c8e188e24bd91ee6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:54:56.887542 23269 catalog_manager.cc:1259] Loaded cluster ID: 3b13775f74b84e0bbcfa8d567f054c0d
I20260925 14:54:56.887626 23269 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: loading cluster ID for follower catalog manager: success
I20260925 14:54:56.889088 23269 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: acquiring CA information for follower catalog manager: success
I20260925 14:54:56.889495 23269 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94ac2c3de8ba496696c85dc0fc6ffcdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:54:56.967110 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:56.963863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.98.61:40853)
0925 14:54:56.963979 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.963982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.963997 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:54:56.964104 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.964106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.964176 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.964292 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:56.964459 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.965002 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.966163 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.966423 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.966508 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:56.966718 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:56.966850 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:56.966945 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:56.966969 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:54:56.966998 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260925 14:54:56.971980 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:56.969425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.0.98.60:46383)
0925 14:54:56.969541 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.969543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.969549 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:54:56.969585 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.969585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.969620 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.969719 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:56.969824 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.970323 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.971155 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.971357 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.971442 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:56.971622 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:56.971746 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:56.971837 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:56.971860 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 14:54:56.971881 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260925 14:54:56.976491 23347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:56.973910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.98.60:46383)
0925 14:54:56.973946 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.973948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.974166 (+ 218us) server_negotiation.cc:407] Connection header received
0925 14:54:56.974265 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.974266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.974277 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.974324 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:56.974470 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.974989 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.975823 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.976023 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.976096 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:56.976251 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:56.976281 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:56.976306 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:56.976401 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:54:56.976420 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:54:56.976749 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:56.973724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.98.61:40853)
0925 14:54:56.973833 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.973835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.973847 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:54:56.973893 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.973894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.973907 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.973955 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:56.974094 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.974832 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.975868 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.976177 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.976232 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:56.976397 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:56.976430 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:56.976490 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:56.976654 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:54:56.976679 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 14:54:56.977429 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:54:56.973736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.0.98.62:45869)
0925 14:54:56.973837 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 14:54:56.973839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:54:56.974087 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:54:56.974178 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:54:56.974179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:54:56.974225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:54:56.974304 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:54:56.974503 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.975245 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.976199 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:54:56.976498 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:54:56.976562 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:54:56.977057 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:54:56.977181 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:54:56.977243 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:54:56.977327 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:54:56.977347 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260925 14:54:56.987449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 22988
I20260925 14:54:56.992663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 23063
I20260925 14:54:56.997360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbEnjDj/build/release/bin/kudu with pid 23138
2026-09-25T14:54:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2795 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5568 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271942 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST